-- Logs begin at Sun 2022-09-25 03:56:06 UTC, end at Sun 2022-09-25 05:00:06 UTC. -- Sep 25 04:21:16.485221 dv-cinder-855359-5-iscsi systemd[1]: Starting Devstack devstack@keystone.service... Sep 25 04:21:16.494285 dv-cinder-855359-5-iscsi devstack@keystone.service[81422]: [uWSGI] getting INI configuration from /etc/keystone/keystone-uwsgi-public.ini Sep 25 04:21:16.496301 dv-cinder-855359-5-iscsi devstack@keystone.service[81422]: *** Starting uWSGI 2.0.18-debian (64bit) on [Sun Sep 25 04:21:16 2022] *** Sep 25 04:21:16.496301 dv-cinder-855359-5-iscsi devstack@keystone.service[81422]: compiled with version: 10.0.1 20200405 (experimental) [master revision 0be9efad938:fcb98e4978a:705510a708d3642c9c962beb663c476167e4e8a4] on 11 April 2020 11:15:55 Sep 25 04:21:16.496301 dv-cinder-855359-5-iscsi devstack@keystone.service[81422]: os: Linux-5.4.0-72-generic #80-Ubuntu SMP Mon Apr 12 17:35:00 UTC 2021 Sep 25 04:21:16.496301 dv-cinder-855359-5-iscsi devstack@keystone.service[81422]: nodename: dv-cinder-855359-5-iscsi Sep 25 04:21:16.496301 dv-cinder-855359-5-iscsi devstack@keystone.service[81422]: machine: x86_64 Sep 25 04:21:16.496301 dv-cinder-855359-5-iscsi devstack@keystone.service[81422]: clock source: unix Sep 25 04:21:16.496301 dv-cinder-855359-5-iscsi devstack@keystone.service[81422]: pcre jit disabled Sep 25 04:21:16.496301 dv-cinder-855359-5-iscsi devstack@keystone.service[81422]: detected number of CPU cores: 4 Sep 25 04:21:16.496301 dv-cinder-855359-5-iscsi devstack@keystone.service[81422]: current working directory: / Sep 25 04:21:16.496301 dv-cinder-855359-5-iscsi devstack@keystone.service[81422]: detected binary path: /usr/bin/uwsgi-core Sep 25 04:21:16.496301 dv-cinder-855359-5-iscsi devstack@keystone.service[81422]: your processes number limit is 47680 Sep 25 04:21:16.496301 dv-cinder-855359-5-iscsi devstack@keystone.service[81422]: your memory page size is 4096 bytes Sep 25 04:21:16.497010 dv-cinder-855359-5-iscsi devstack@keystone.service[81422]: detected max file descriptor number: 2048 Sep 25 04:21:16.497010 dv-cinder-855359-5-iscsi devstack@keystone.service[81422]: lock engine: pthread robust mutexes Sep 25 04:21:16.497010 dv-cinder-855359-5-iscsi devstack@keystone.service[81422]: thunder lock: enabled Sep 25 04:21:16.497010 dv-cinder-855359-5-iscsi devstack@keystone.service[81422]: uwsgi socket 0 bound to UNIX address /var/run/uwsgi/keystone-wsgi-public.socket fd 4 Sep 25 04:21:16.497010 dv-cinder-855359-5-iscsi devstack@keystone.service[81422]: Python version: 3.8.10 (default, Jun 22 2022, 20:18:18) [GCC 9.4.0] Sep 25 04:21:16.535695 dv-cinder-855359-5-iscsi devstack@keystone.service[81422]: Python main interpreter initialized at 0x56068a1449c0 Sep 25 04:21:16.535695 dv-cinder-855359-5-iscsi devstack@keystone.service[81422]: python threads support enabled Sep 25 04:21:16.535695 dv-cinder-855359-5-iscsi devstack@keystone.service[81422]: your server socket listen backlog is limited to 100 connections Sep 25 04:21:16.535695 dv-cinder-855359-5-iscsi devstack@keystone.service[81422]: your mercy for graceful operations on workers is 90 seconds Sep 25 04:21:16.536062 dv-cinder-855359-5-iscsi devstack@keystone.service[81422]: mapped 403077 bytes (393 KB) for 2 cores Sep 25 04:21:16.536143 dv-cinder-855359-5-iscsi devstack@keystone.service[81422]: *** Operational MODE: preforking *** Sep 25 04:21:16.536387 dv-cinder-855359-5-iscsi devstack@keystone.service[81422]: *** uWSGI is running in multiple interpreter mode *** Sep 25 04:21:16.536387 dv-cinder-855359-5-iscsi devstack@keystone.service[81422]: spawned uWSGI master process (pid: 81422) Sep 25 04:21:16.536587 dv-cinder-855359-5-iscsi systemd[1]: Started Devstack devstack@keystone.service. Sep 25 04:21:16.537714 dv-cinder-855359-5-iscsi devstack@keystone.service[81422]: spawned uWSGI worker 1 (pid: 81433, cores: 1) Sep 25 04:21:16.537714 dv-cinder-855359-5-iscsi devstack@keystone.service[81422]: spawned uWSGI worker 2 (pid: 81434, cores: 1) Sep 25 04:21:16.537714 dv-cinder-855359-5-iscsi devstack@keystone.service[81422]: running "unix_signal:15 gracefully_kill_them_all" (master-start)... Sep 25 04:21:20.832336 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Sep 25 04:21:20.832336 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: from cryptography.utils import int_from_bytes Sep 25 04:21:20.833154 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Sep 25 04:21:20.833154 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: from cryptography.utils import int_from_bytes Sep 25 04:21:20.833280 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Sep 25 04:21:20.833280 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: from cryptography.utils import int_from_bytes Sep 25 04:21:20.833695 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Sep 25 04:21:20.833695 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: from cryptography.utils import int_from_bytes Sep 25 04:21:20.882538 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Sep 25 04:21:20.882713 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Sep 25 04:21:20.882807 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Sep 25 04:21:20.882898 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Sep 25 04:21:20.882979 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] command line args: [] {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Sep 25 04:21:20.883058 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] command line args: [] {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Sep 25 04:21:20.883165 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Sep 25 04:21:20.883258 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Sep 25 04:21:20.883338 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ================================================================================ {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Sep 25 04:21:20.883411 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ================================================================================ {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Sep 25 04:21:20.883592 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] admin_token = **** {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 04:21:20.883678 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] admin_token = **** {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 04:21:20.883843 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] config_dir = [] {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 04:21:20.883931 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] config_dir = [] {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 04:21:20.884035 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 04:21:20.884123 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 04:21:20.884203 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] config_source = [] {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 04:21:20.884275 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] config_source = [] {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 04:21:20.884393 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] debug = True {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 04:21:20.884482 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] debug = True {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 04:21:20.884593 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 04:21:20.884930 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 04:21:20.885071 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 04:21:20.885178 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 04:21:20.885276 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] insecure_debug = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 04:21:20.885370 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] insecure_debug = False {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 04:21:20.885481 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 04:21:20.885578 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 04:21:20.885685 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 04:21:20.885783 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 04:21:20.885885 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] list_limit = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 04:21:20.885986 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] list_limit = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 04:21:20.886083 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] log_config_append = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 04:21:20.886178 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] log_config_append = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 04:21:20.886282 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 04:21:20.886386 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 04:21:20.886481 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] log_dir = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 04:21:20.886575 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] log_dir = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 04:21:20.886670 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] log_file = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 04:21:20.886777 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] log_file = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 04:21:20.886905 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 04:21:20.886989 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 04:21:20.887113 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 04:21:20.887196 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 04:21:20.887275 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 04:21:20.887393 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 04:21:20.887479 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 04:21:20.887556 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 04:21:20.887682 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 04:21:20.887766 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 04:21:20.887840 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 04:21:20.887966 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 04:21:20.888044 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 04:21:20.888167 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 04:21:20.888274 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 04:21:20.888355 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 04:21:20.888512 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 04:21:20.888599 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 04:21:20.888777 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 04:21:20.888859 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 04:21:20.889016 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 04:21:20.889096 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 04:21:20.889253 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 04:21:20.889380 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 04:21:20.889502 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 04:21:20.889597 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 04:21:20.889753 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] notification_format = cadf {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 04:21:20.889830 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] notification_format = cadf {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 04:21:20.890032 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 04:21:20.890122 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 04:21:20.890529 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.25/identity {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 04:21:20.890609 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.25/identity {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 04:21:20.890739 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] publish_errors = False {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 04:21:20.890816 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] publish_errors = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 04:21:20.890938 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 04:21:20.891021 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 04:21:20.891161 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 04:21:20.891252 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 04:21:20.891326 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 04:21:20.891443 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 04:21:20.891578 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 04:21:20.891679 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 04:21:20.891816 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 04:21:20.891898 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 04:21:20.892052 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] standard_threads = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 04:21:20.892130 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] standard_threads = False {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 04:21:20.892285 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] strict_password_check = False {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 04:21:20.892371 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] strict_password_check = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 04:21:20.892533 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 04:21:20.892612 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 04:21:20.892719 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] use_eventlog = False {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 04:21:20.892799 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] use_eventlog = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 04:21:20.892904 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] use_journal = False {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 04:21:20.893035 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] use_journal = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 04:21:20.893112 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] use_json = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 04:21:20.893183 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] use_json = False {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 04:21:20.893261 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] use_stderr = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 04:21:20.893340 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] use_stderr = False {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 04:21:20.893418 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] use_syslog = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 04:21:20.893499 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] use_syslog = False {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 04:21:20.893628 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] watch_log_file = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 04:21:20.894328 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] watch_log_file = False {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 25 04:21:20.894451 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.894526 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.894694 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.894784 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.894963 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.895049 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.895209 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.895288 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.895467 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.895556 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.895708 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.895786 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.895961 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.896038 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.896199 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.896289 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.896452 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.896536 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.896726 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.896820 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.896982 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.897071 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.897221 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.897302 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.897472 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.897550 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.897712 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.897812 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.897964 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.898043 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.898217 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.898295 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.898457 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.898545 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.898716 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.898794 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.898975 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.899060 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.899223 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.899384 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.899461 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.899532 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.899710 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.899790 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.899973 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.900058 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.900231 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.900321 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.900484 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.900562 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.900743 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.900822 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.900983 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.901060 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.901219 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.901298 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.901457 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.901536 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.901710 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.901787 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.902000 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.902089 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.902196 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.902282 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.902431 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.902516 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.902681 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.902760 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.902934 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.903013 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.903175 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.903252 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.903428 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.903511 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.903663 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.903743 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.903901 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.903985 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.904142 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.904220 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.904375 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.904455 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.904609 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.904728 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.904868 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.904958 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.905129 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.905263 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.905338 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.905410 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.905571 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.905653 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.905804 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.905884 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.906042 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.906122 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.906277 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.906354 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.906513 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.906593 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.906750 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.906827 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.907002 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.907079 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.907237 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.907347 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.907476 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.907579 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.907714 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.907819 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.907952 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.908057 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.908189 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.908344 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.908432 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.908505 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.908661 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.908756 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.908917 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.908996 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.909155 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.909244 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.909942 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.910028 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.910190 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.910284 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.910457 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.910553 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.910699 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.910794 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.910939 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.911064 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.911181 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.911261 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.911425 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.911512 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.911701 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.911780 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.911944 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.912029 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.912197 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.912283 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.912448 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.912527 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.912718 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.912845 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.912962 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.913053 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.913201 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.913283 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.913452 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] cache.enabled = True {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.913532 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] cache.enabled = True {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.913691 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.913777 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.913932 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.914011 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.914178 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.914255 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.914420 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.914510 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.914672 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.914754 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.914913 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.914991 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.915149 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.915238 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.915388 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.915466 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.915623 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.915701 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.915860 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.915946 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.916124 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.916209 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.916369 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.916448 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.916622 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.916764 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.916890 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.916971 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.917132 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.917210 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.917378 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.917533 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.917611 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.917682 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.917847 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.917928 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.918083 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.918171 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.918318 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.918398 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.918552 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.918633 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.918787 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.918865 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.919022 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.919109 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.919257 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.919341 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.919500 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.919579 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.919741 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.919826 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.919994 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.920073 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.920236 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.920314 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.920499 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.920586 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.920792 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.920874 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.921040 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.921157 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.921277 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] auth.external = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.921358 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] auth.external = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.921513 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] auth.mapped = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.921594 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] auth.mapped = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.921814 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.921894 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.922052 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.922131 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.922288 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] auth.password = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.922368 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] auth.password = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.922524 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] auth.token = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.922610 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] auth.token = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.922768 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.922848 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.923008 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] catalog.caching = True {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.923089 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] catalog.caching = True {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.923259 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.923338 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.923492 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.923571 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.923744 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.923825 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.923994 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.924073 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.924227 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.924307 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.924466 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] credential.caching = True {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.924546 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] credential.caching = True {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.924744 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] credential.driver = sql {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.924829 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] credential.driver = sql {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.925026 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.925114 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.925282 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.925365 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.925541 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.925622 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.925791 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.925872 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.926032 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.926122 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.926285 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.926366 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.926545 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.926627 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.926785 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.926868 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.927046 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.927125 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.927331 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.927410 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.927578 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.927659 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.927842 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.927921 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.928087 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.928167 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.928349 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.928438 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.928587 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] federation.caching = True {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.928681 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] federation.caching = True {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.928855 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.928937 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.929110 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] federation.driver = sql {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.929195 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] federation.driver = sql {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.929363 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.929442 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.929598 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.929678 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.929852 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.929934 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.930092 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.930173 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.930350 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.930438 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.930591 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.930671 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.930851 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.930932 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.931091 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.931179 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.931340 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.931420 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.931579 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] identity.caching = True {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.931659 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] identity.caching = True {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.931833 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.931991 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.932070 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.932142 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.932314 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.932396 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.932552 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.932661 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.932830 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] identity.driver = sql {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.932910 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] identity.driver = sql {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.933065 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.933145 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.933307 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.933396 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.933560 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.933640 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.933814 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.933894 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.934049 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.934142 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.934284 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.934363 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.934517 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.934596 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.934763 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.934851 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.935014 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.935094 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.935268 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.935347 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.935527 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.935630 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.935784 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.935904 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.936064 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.936145 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.936306 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.936397 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.936548 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.936626 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.936809 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.936888 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.937052 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.937139 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.937289 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.937366 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.937530 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.937607 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.937774 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.937860 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.938017 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.938094 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.938271 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.938349 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.938504 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.938589 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.938760 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.938837 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.939014 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.939092 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.939252 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.939337 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.939505 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.939582 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.939760 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.939836 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.939992 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.940069 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.940235 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.940312 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.940471 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.password = **** {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.940605 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.password = **** {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.940820 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.940907 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.941000 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.941118 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.941242 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.941317 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.941486 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.941564 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.941728 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.941805 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.941981 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.942058 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.942227 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.942342 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.942472 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.942581 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.942713 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.942813 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.943079 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.943157 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.943334 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.943414 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.943576 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.943655 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.943810 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.943892 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.944054 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.944140 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.944295 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.user = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.944372 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.user = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.944540 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.944617 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.944831 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.944910 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.945068 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.945144 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.945325 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.945404 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.945579 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.945655 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.945825 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.945900 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.946068 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.946145 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.946306 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.946384 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.946547 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.946624 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.946785 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.946862 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.947026 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.947103 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.947261 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.947338 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.947514 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.947591 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.947755 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.947837 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.948012 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.948088 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.948253 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.948329 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.948507 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.948585 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.948766 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.948850 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.949019 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.949096 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.949258 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.949335 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.949497 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.949573 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.949738 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.949815 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.950220 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.950298 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.950478 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.950556 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.950727 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.950805 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.950970 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.951049 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.951229 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] policy.driver = sql {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.951306 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] policy.driver = sql {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.951467 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.951544 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.951713 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.951789 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.951950 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.952032 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.952191 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] receipt.caching = True {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.952269 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] receipt.caching = True {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.952428 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.952512 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.952699 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.952778 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.952951 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.953028 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.953187 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.953265 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.953424 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.953502 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.953671 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] resource.caching = True {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.953748 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] resource.caching = True {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.953924 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.954002 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.954175 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] resource.driver = sql {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.954250 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] resource.driver = sql {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.954503 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.954582 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.954644 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.954703 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.955047 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.955123 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.955123 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] revoke.caching = True {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.955250 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] revoke.caching = True {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.955585 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.955661 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.955661 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.955769 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.955905 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] role.cache_time = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.955970 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] role.cache_time = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.956697 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] role.caching = True {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.956797 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] role.caching = True {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.956864 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] role.driver = sql {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.956934 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] role.driver = sql {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.957074 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] role.list_limit = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.957187 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] role.list_limit = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.957329 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.957478 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.957585 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.957679 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.957839 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.957921 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.958092 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.958174 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.958343 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.958433 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.958594 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.958674 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.958844 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.958931 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.959098 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.959187 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.959334 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.959414 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.959586 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.959668 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.959837 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.959926 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.960090 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.960171 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.960342 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.960423 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.960766 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.960863 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.961008 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.961088 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.961264 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.961344 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.961520 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.961609 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.961774 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.961855 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.962026 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.962106 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.962266 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.962354 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.962523 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.962604 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.962777 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.962858 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.963018 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.963100 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.963254 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.963334 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.963490 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.963571 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.963727 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.963809 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.963979 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.964059 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.964238 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.964318 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.964488 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.964570 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.964751 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.964833 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.964992 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.965072 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.965227 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] token.cache_time = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.965309 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] token.cache_time = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.965465 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] token.caching = True {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.965545 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] token.caching = True {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.965705 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.965785 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.965957 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] token.provider = fernet {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.966039 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] token.provider = fernet {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.966196 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.966276 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.966455 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.966537 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.966706 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.966788 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.966945 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.967027 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.967192 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.967272 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.967436 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.967517 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.967686 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] trust.driver = sql {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.967767 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] trust.driver = sql {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.967926 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.968006 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.968165 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.968246 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.968402 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.968482 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.968670 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.968775 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.968956 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.969047 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.969189 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.969287 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.969435 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.969532 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.969704 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.969791 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.969967 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] database.connection = **** {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.970062 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] database.connection = **** {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.970212 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.970299 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.970465 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.970552 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.970705 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.970805 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.970949 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.971037 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.971185 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.971272 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.971426 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.971521 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.971661 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.971749 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.971946 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.972033 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.972197 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.972294 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.972435 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.972522 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.972704 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.972794 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.972948 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.973050 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.973206 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.973295 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.973447 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.973536 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.973699 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.973796 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.973939 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.974027 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.974180 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.974267 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.974418 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.974514 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 25 04:21:20.974591 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Sep 25 04:21:20.974663 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81434) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Sep 25 04:21:21.004861 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=81433) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 04:21:21.005093 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=81433) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 04:21:21.005239 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=81434) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 04:21:21.005365 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=81433) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 04:21:21.005441 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=81434) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 04:21:21.005671 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=81434) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 04:21:21.013611 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=81433) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 04:21:21.013980 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=81434) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 04:21:21.042295 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=81433) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 04:21:21.042428 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=81434) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 04:21:21.042506 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=81433) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 04:21:21.042581 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=81434) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 04:21:21.097277 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=81434) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 04:21:21.097711 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=81433) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 04:21:21.146614 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=81434) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 04:21:21.146761 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=81433) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 04:21:21.146848 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=81434) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 04:21:21.146928 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=81433) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 04:21:21.302562 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81433) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 04:21:21.302749 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81434) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 04:21:21.302837 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=81433) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 04:21:21.302915 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=81434) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 04:21:21.302992 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=81433) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 04:21:21.303091 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=81434) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 04:21:21.303262 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=81433) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 04:21:21.303358 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=81434) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 04:21:21.303432 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=81433) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 04:21:21.303505 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=81434) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 04:21:21.303648 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=81433) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 04:21:21.303740 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=81434) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 04:21:21.326628 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81433) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 04:21:21.326851 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81434) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 04:21:21.326978 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=81433) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 04:21:21.327100 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=81433) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 04:21:21.327204 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=81434) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 04:21:21.327283 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=81433) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 04:21:21.327357 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=81434) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 04:21:21.327509 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=81433) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 04:21:21.327600 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=81434) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 04:21:21.327742 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=81433) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 04:21:21.327833 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=81434) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 04:21:21.327955 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81433) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 04:21:21.328044 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=81434) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 04:21:21.328157 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81434) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 04:21:21.339309 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=81434) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 04:21:21.339501 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=81433) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 04:21:21.352668 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81434) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 04:21:21.352844 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81433) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 04:21:21.352953 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=81434) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 04:21:21.353093 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=81434) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 04:21:21.353183 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=81433) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 04:21:21.353297 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=81434) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 04:21:21.353385 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=81433) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 04:21:21.353499 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=81434) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 04:21:21.353586 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=81433) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 04:21:21.353710 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=81433) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 04:21:21.353797 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=81434) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 04:21:21.353916 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81434) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 04:21:21.354003 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=81433) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 04:21:21.354127 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81433) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 25 04:21:21.371294 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.371510 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Sep 25 04:21:21.371683 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.371858 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.371986 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Sep 25 04:21:21.372115 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Sep 25 04:21:21.372252 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.372379 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.372503 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Sep 25 04:21:21.372626 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.372782 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.372933 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.373064 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.373192 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.373321 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.373449 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.382806 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.383094 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.383225 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.383377 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.383505 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.383654 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.383799 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.383927 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.390655 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=81434) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 25 04:21:21.390858 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=81433) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 25 04:21:21.393573 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=81434) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 25 04:21:21.393804 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=81433) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 25 04:21:21.393941 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.394071 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.394294 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.394420 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.394587 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.394732 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.394865 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.394994 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.395122 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.395250 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.395386 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.395513 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.395642 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.395769 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.395899 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.396045 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.396175 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.396303 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.396430 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.396558 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.419338 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.419497 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.420837 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=81433) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 25 04:21:21.420970 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=81434) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 25 04:21:21.421126 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.421252 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.425854 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=81433) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 25 04:21:21.426005 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=81434) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 25 04:21:21.426132 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.426257 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.426414 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.426553 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.432695 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=81433) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 25 04:21:21.432825 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=81434) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 25 04:21:21.432989 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.433114 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.438552 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=81433) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 25 04:21:21.438722 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=81434) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 25 04:21:21.438849 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.438974 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.439131 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.439258 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.439424 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.439551 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.439707 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.439833 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.439983 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.440116 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.440251 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.440375 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.441041 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.441172 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.457220 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.457397 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.457524 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.457683 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.461342 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=81433) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 25 04:21:21.461470 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=81434) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 25 04:21:21.464542 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.464708 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.464865 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.464993 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.469060 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=81433) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 25 04:21:21.469243 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=81434) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 25 04:21:21.472145 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=81434) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 25 04:21:21.472279 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=81433) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 25 04:21:21.477140 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.477317 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.477487 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.477626 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.477759 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.477892 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.478023 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.478163 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.478295 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.478426 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.478556 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.478687 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.492461 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=81433) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 25 04:21:21.492604 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=81434) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 25 04:21:21.492781 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.492908 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.493039 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.493179 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.493307 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.493436 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.500085 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH2: ['/token' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.500225 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH2: ['/token' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.501644 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.501828 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.503716 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.504013 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.504199 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.504497 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.506829 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=81434) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 25 04:21:21.507123 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.507250 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=81433) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 25 04:21:21.507411 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.507539 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.507699 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.507838 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.507964 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.508097 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.508358 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.519815 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=81434) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 25 04:21:21.520050 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=81433) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 25 04:21:21.520179 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.520342 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.520469 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.520611 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.520772 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.520927 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.521053 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.521315 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.521452 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.521610 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.521736 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.521895 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.536942 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=81434) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 25 04:21:21.537109 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=81433) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 25 04:21:21.539845 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=81434) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 25 04:21:21.539974 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=81433) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 25 04:21:21.542747 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.542922 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.544223 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.544352 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.545724 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=81434) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 25 04:21:21.545856 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=81433) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 25 04:21:21.546014 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.546141 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.546414 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.546552 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.553348 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.553494 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.554821 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=81433) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 25 04:21:21.554964 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=81434) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 25 04:21:21.557700 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.557834 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.558098 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.558224 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.558391 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.558526 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.558667 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.558874 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.566895 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=81433) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 25 04:21:21.567044 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=81434) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 25 04:21:21.567184 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.567310 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.567465 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.567593 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.574294 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=81433) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 25 04:21:21.574442 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=81434) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 25 04:21:21.574576 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.574718 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.574850 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.575225 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.575384 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.575526 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.575658 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.575789 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.575915 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.576039 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.576173 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.576327 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.576470 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.576597 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.576759 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.576890 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.577020 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.577149 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.577283 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.577418 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.577546 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.577675 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.577803 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.577958 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.578087 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.578223 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=81434) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 25 04:21:21.611652 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: WSGI app 0 (mountpoint='') ready in 5 seconds on interpreter 0x56068a1449c0 pid: 81433 (default app) Sep 25 04:21:21.611913 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: WSGI app 0 (mountpoint='') ready in 5 seconds on interpreter 0x56068a1449c0 pid: 81434 (default app) Sep 25 04:21:21.615532 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b45cf65-2ae4-424e-928c-6fc224255228 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:21.615822 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b45cf65-2ae4-424e-928c-6fc224255228 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:21.616099 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b45cf65-2ae4-424e-928c-6fc224255228 None None] PATH_INFO: `/v3` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:21.620249 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1/1] 10.222.0.25 () {50 vars in 791 bytes} [Sun Sep 25 04:21:21 2022] GET /identity/v3/ => generated 255 bytes in 8 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:21:28.421890 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f12621c-b8f9-4360-9432-767b664e6f2c None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:28.422416 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f12621c-b8f9-4360-9432-767b664e6f2c None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:28.422578 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f12621c-b8f9-4360-9432-767b664e6f2c None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:28.428328 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1/2] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:21:28 2022] GET /identity => generated 270 bytes in 10 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 25 04:21:28.434100 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0d4023b-e1b5-4bf2-bbf9-a4758d95d2ce None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:28.434578 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0d4023b-e1b5-4bf2-bbf9-a4758d95d2ce None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:28.434718 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0d4023b-e1b5-4bf2-bbf9-a4758d95d2ce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:28.514606 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: INFO dbcounter [None req-b0d4023b-e1b5-4bf2-bbf9-a4758d95d2ce None None] Registered counter for database keystone Sep 25 04:21:28.535952 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG oslo_db.sqlalchemy.engines [None req-b0d4023b-e1b5-4bf2-bbf9-a4758d95d2ce None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=81433) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Sep 25 04:21:28.542373 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writer thread running {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Sep 25 04:21:28.745071 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG passlib.handlers.bcrypt [None req-b0d4023b-e1b5-4bf2-bbf9-a4758d95d2ce None None] detected 'bcrypt' backend, version '4.0.0' {{(pid=81433) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Sep 25 04:21:28.745385 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG passlib.handlers.bcrypt [None req-b0d4023b-e1b5-4bf2-bbf9-a4758d95d2ce None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=81433) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Sep 25 04:21:28.808810 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-b0d4023b-e1b5-4bf2-bbf9-a4758d95d2ce None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:21:28.810467 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-b0d4023b-e1b5-4bf2-bbf9-a4758d95d2ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:21:28.965318 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 2/3] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:21:28 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 532 msecs (HTTP/1.1 201) 6 headers in 384 bytes (2 switches on core 0) Sep 25 04:21:29.261048 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f12621c-b8f9-4360-9432-767b664e6f2c None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:21:29.333234 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: INFO dbcounter [None req-8f12621c-b8f9-4360-9432-767b664e6f2c None None] Registered counter for database keystone Sep 25 04:21:29.355828 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG oslo_db.sqlalchemy.engines [None req-8f12621c-b8f9-4360-9432-767b664e6f2c None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=81434) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Sep 25 04:21:29.362088 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writer thread running {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Sep 25 04:21:29.482128 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f12621c-b8f9-4360-9432-767b664e6f2c None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:21:29.486430 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc4f69ab-18cb-4ba4-a379-bc10481e4403 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:21:29.487098 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc4f69ab-18cb-4ba4-a379-bc10481e4403 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:29.487419 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc4f69ab-18cb-4ba4-a379-bc10481e4403 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:29.487664 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc4f69ab-18cb-4ba4-a379-bc10481e4403 admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:29.495967 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc4f69ab-18cb-4ba4-a379-bc10481e4403 admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:21:29.554811 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: WARNING keystone.common.rbac_enforcer.enforcer [None req-bc4f69ab-18cb-4ba4-a379-bc10481e4403 admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Sep 25 04:21:29.555099 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc4f69ab-18cb-4ba4-a379-bc10481e4403 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:21:29.564775 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 2/4] 10.222.0.25 () {58 vars in 1324 bytes} [Sun Sep 25 04:21:29 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 304 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 25 04:21:29.571406 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0d4023b-e1b5-4bf2-bbf9-a4758d95d2ce None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:21:29.589120 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0d4023b-e1b5-4bf2-bbf9-a4758d95d2ce None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:21:29.593598 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f2cb6f7-3a26-4442-a099-f3bb7b8be48b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:21:29.594710 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f2cb6f7-3a26-4442-a099-f3bb7b8be48b admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:29.594996 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f2cb6f7-3a26-4442-a099-f3bb7b8be48b admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:29.595295 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f2cb6f7-3a26-4442-a099-f3bb7b8be48b admin admin] PATH_INFO: `/v3/projects/99bb568eb0e94976a15231a76508d22f` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:29.597051 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f2cb6f7-3a26-4442-a099-f3bb7b8be48b admin admin] RBAC: Authorizing `identity:get_project(project_id=99bb568eb0e94976a15231a76508d22f)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:21:29.650972 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: WARNING keystone.common.rbac_enforcer.enforcer [None req-7f2cb6f7-3a26-4442-a099-f3bb7b8be48b admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Sep 25 04:21:29.651323 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f2cb6f7-3a26-4442-a099-f3bb7b8be48b admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:21:29.655850 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 3/5] 10.222.0.25 () {56 vars in 1207 bytes} [Sun Sep 25 04:21:29 2022] GET /identity/v3/projects/99bb568eb0e94976a15231a76508d22f => generated 344 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:21:31.776930 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-904b9c53-fd25-4f0c-84d7-d7015869c73c None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:31.777465 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-904b9c53-fd25-4f0c-84d7-d7015869c73c None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:31.777465 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-904b9c53-fd25-4f0c-84d7-d7015869c73c None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:31.781515 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 3/6] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:21:31 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 25 04:21:31.787382 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4f7476c-6df5-4cef-aebd-64ae876a2dad None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:31.787773 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4f7476c-6df5-4cef-aebd-64ae876a2dad None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:31.787937 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4f7476c-6df5-4cef-aebd-64ae876a2dad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:31.808189 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-d4f7476c-6df5-4cef-aebd-64ae876a2dad None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:21:31.809745 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-d4f7476c-6df5-4cef-aebd-64ae876a2dad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:21:31.817613 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 4/7] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:21:31 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Sep 25 04:21:32.085798 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-904b9c53-fd25-4f0c-84d7-d7015869c73c None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:21:32.098372 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-904b9c53-fd25-4f0c-84d7-d7015869c73c None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:21:32.102630 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a072ea78-f809-401d-81cc-9166c8a51bfd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:21:32.103230 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a072ea78-f809-401d-81cc-9166c8a51bfd admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:32.103509 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a072ea78-f809-401d-81cc-9166c8a51bfd admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:32.103779 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a072ea78-f809-401d-81cc-9166c8a51bfd admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:32.111305 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a072ea78-f809-401d-81cc-9166c8a51bfd admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:21:32.113157 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a072ea78-f809-401d-81cc-9166c8a51bfd admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:21:32.121740 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 4/8] 10.222.0.25 () {58 vars in 1324 bytes} [Sun Sep 25 04:21:32 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 25 04:21:32.127156 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4f7476c-6df5-4cef-aebd-64ae876a2dad None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:21:32.136455 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4f7476c-6df5-4cef-aebd-64ae876a2dad None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:21:32.140901 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af2648cd-4699-4d96-bda1-2ecab4c70242 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:21:32.141534 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af2648cd-4699-4d96-bda1-2ecab4c70242 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:32.141822 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af2648cd-4699-4d96-bda1-2ecab4c70242 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:32.142101 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af2648cd-4699-4d96-bda1-2ecab4c70242 admin admin] PATH_INFO: `/v3/users/e61aa4e8d68f455e96668bc70e0a6679` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:32.144103 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af2648cd-4699-4d96-bda1-2ecab4c70242 admin admin] RBAC: Authorizing `identity:get_user(user_id=e61aa4e8d68f455e96668bc70e0a6679)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:21:32.146135 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af2648cd-4699-4d96-bda1-2ecab4c70242 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:21:32.147646 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 5/9] 10.222.0.25 () {56 vars in 1198 bytes} [Sun Sep 25 04:21:32 2022] GET /identity/v3/users/e61aa4e8d68f455e96668bc70e0a6679 => generated 247 bytes in 21 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:21:34.376694 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87ddf61f-2610-48ef-93dc-fd53b34d0c90 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:34.377261 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87ddf61f-2610-48ef-93dc-fd53b34d0c90 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:34.377261 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87ddf61f-2610-48ef-93dc-fd53b34d0c90 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:34.378757 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 5/10] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:21:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:21:34.384162 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f93b84a-5214-4837-8ff4-73c6b9ad303b None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:34.384572 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f93b84a-5214-4837-8ff4-73c6b9ad303b None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:34.384816 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f93b84a-5214-4837-8ff4-73c6b9ad303b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:34.388018 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43fa6577-6d87-4524-a034-f3c23ceff98f None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:34.388302 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43fa6577-6d87-4524-a034-f3c23ceff98f None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:34.388579 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43fa6577-6d87-4524-a034-f3c23ceff98f None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:34.389838 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 6/11] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:21:34 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 25 04:21:34.395411 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d21f95c-c15f-4120-860a-94327a5e481a None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:34.395700 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d21f95c-c15f-4120-860a-94327a5e481a None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:34.395978 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d21f95c-c15f-4120-860a-94327a5e481a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:34.403917 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-1f93b84a-5214-4837-8ff4-73c6b9ad303b None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:21:34.405125 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-1f93b84a-5214-4837-8ff4-73c6b9ad303b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:21:34.437850 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG passlib.handlers.bcrypt [None req-7d21f95c-c15f-4120-860a-94327a5e481a None None] detected 'bcrypt' backend, version '4.0.0' {{(pid=81434) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Sep 25 04:21:34.438245 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG passlib.handlers.bcrypt [None req-7d21f95c-c15f-4120-860a-94327a5e481a None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=81434) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Sep 25 04:21:34.487695 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 6/12] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:21:34 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 04:21:34.487879 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-7d21f95c-c15f-4120-860a-94327a5e481a None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:21:34.489535 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-7d21f95c-c15f-4120-860a-94327a5e481a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:21:34.589494 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 7/13] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:21:34 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 195 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 25 04:21:34.757222 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f93b84a-5214-4837-8ff4-73c6b9ad303b None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:21:34.781145 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:21:34.861228 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d21f95c-c15f-4120-860a-94327a5e481a None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:21:34.866569 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f93b84a-5214-4837-8ff4-73c6b9ad303b None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:21:34.900377 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e2e3411-cf6c-477f-91b7-7cf1cb4cc344 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:21:34.901044 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e2e3411-cf6c-477f-91b7-7cf1cb4cc344 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:34.901327 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e2e3411-cf6c-477f-91b7-7cf1cb4cc344 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:34.901602 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e2e3411-cf6c-477f-91b7-7cf1cb4cc344 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:34.913171 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e2e3411-cf6c-477f-91b7-7cf1cb4cc344 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 04:21:34.922112 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-7e2e3411-cf6c-477f-91b7-7cf1cb4cc344 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 04:21:34.922112 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:21:34.922112 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:21:34.922112 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:21:34.922112 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:21:34.922112 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:21:34.922112 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:21:34.922112 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:21:34.922112 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:21:34.922112 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:21:34.922112 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:21:34.922112 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:21:34.922112 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 04:21:34.922112 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 04:21:34.922112 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 04:21:34.922112 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 04:21:34.923213 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 04:21:34.923213 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 04:21:34.923213 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 04:21:34.923213 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 04:21:34.923213 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:21:34.923213 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:21:34.923213 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:21:34.923213 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:21:34.923213 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:21:34.923213 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:21:34.923213 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:21:34.923213 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:21:34.923213 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:21:34.923213 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:21:34.923213 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:21:34.923213 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:21:34.924280 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:21:34.924280 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:21:34.924280 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:21:34.924280 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:21:34.924280 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 04:21:34.924280 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 04:21:34.924280 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 04:21:34.924280 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 04:21:34.924280 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 04:21:34.924280 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 04:21:34.924280 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 04:21:34.924280 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:21:34.924280 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 7/14] 10.222.0.25 () {56 vars in 1096 bytes} [Sun Sep 25 04:21:34 2022] GET /identity/v3/roles/admin => generated 83 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:21:34.929499 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e2e3411-cf6c-477f-91b7-7cf1cb4cc344 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:21:34.965085 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d21f95c-c15f-4120-860a-94327a5e481a None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:21:34.967326 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:21:35.002045 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89b046da-a655-4273-9d36-d6b45ef569cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:21:35.003414 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b046da-a655-4273-9d36-d6b45ef569cb None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:35.003691 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b046da-a655-4273-9d36-d6b45ef569cb None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:35.003963 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b046da-a655-4273-9d36-d6b45ef569cb None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:35.012447 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:21:35.031965 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e2e3411-cf6c-477f-91b7-7cf1cb4cc344 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:21:35.066037 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aba6a090-54b0-4126-8915-ac224983dbe5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:21:35.066653 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aba6a090-54b0-4126-8915-ac224983dbe5 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:35.066934 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aba6a090-54b0-4126-8915-ac224983dbe5 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:35.067206 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aba6a090-54b0-4126-8915-ac224983dbe5 None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:35.067881 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aba6a090-54b0-4126-8915-ac224983dbe5 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:21:35.068188 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aba6a090-54b0-4126-8915-ac224983dbe5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:21:35.069613 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aba6a090-54b0-4126-8915-ac224983dbe5 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:21:35.073610 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89b046da-a655-4273-9d36-d6b45ef569cb None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:21:35.075401 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89b046da-a655-4273-9d36-d6b45ef569cb None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:21:35.081934 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 8/15] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Sep 25 04:21:34 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:21:35.087635 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aba6a090-54b0-4126-8915-ac224983dbe5 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:21:35.175424 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 8/16] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Sep 25 04:21:34 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 315 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 25 04:21:35.181009 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89b046da-a655-4273-9d36-d6b45ef569cb None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:21:35.183619 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:21:35.184233 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aba6a090-54b0-4126-8915-ac224983dbe5 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:21:35.219119 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2542aaf-4701-420d-a54e-a20cbc78688b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:21:35.219752 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2542aaf-4701-420d-a54e-a20cbc78688b None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:35.220042 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2542aaf-4701-420d-a54e-a20cbc78688b None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:35.220315 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2542aaf-4701-420d-a54e-a20cbc78688b None admin] PATH_INFO: `/v3/users/e61aa4e8d68f455e96668bc70e0a6679` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:35.222048 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2542aaf-4701-420d-a54e-a20cbc78688b None admin] RBAC: Authorizing `identity:get_user(user_id=e61aa4e8d68f455e96668bc70e0a6679)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:21:35.223499 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2542aaf-4701-420d-a54e-a20cbc78688b None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:21:35.225101 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 9/17] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Sep 25 04:21:35 2022] GET /identity/v3/users/e61aa4e8d68f455e96668bc70e0a6679 => generated 247 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:21:35.230529 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2542aaf-4701-420d-a54e-a20cbc78688b None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:21:35.244496 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:21:35.281379 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89b046da-a655-4273-9d36-d6b45ef569cb None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:21:35.316970 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15137f4b-f2f9-4ce8-810e-88aa7ee5718d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:21:35.317830 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15137f4b-f2f9-4ce8-810e-88aa7ee5718d None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:35.318108 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15137f4b-f2f9-4ce8-810e-88aa7ee5718d None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:35.318386 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15137f4b-f2f9-4ce8-810e-88aa7ee5718d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:35.327871 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2542aaf-4701-420d-a54e-a20cbc78688b None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:21:35.336986 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15137f4b-f2f9-4ce8-810e-88aa7ee5718d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:21:35.338451 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15137f4b-f2f9-4ce8-810e-88aa7ee5718d None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:21:35.356624 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-15137f4b-f2f9-4ce8-810e-88aa7ee5718d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:21:35.356624 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:21:35.356624 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:21:35.356624 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:21:35.356624 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:21:35.356624 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:21:35.356624 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:21:35.356624 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:21:35.356624 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:21:35.356624 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:21:35.356624 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:21:35.356624 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:21:35.356624 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 04:21:35.356624 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 04:21:35.356624 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 04:21:35.356624 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 04:21:35.357546 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:21:35.357546 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:21:35.357546 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:21:35.357546 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:21:35.357546 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:21:35.357546 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:21:35.357546 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:21:35.357546 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:21:35.357546 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:21:35.357546 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:21:35.357546 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:21:35.357546 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:21:35.357546 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:21:35.357546 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:21:35.357546 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:21:35.357546 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:21:35.357546 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 04:21:35.358366 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 04:21:35.358366 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:21:35.358366 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:21:35.358366 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 9/18] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:21:35 2022] GET /identity/v3/domains/Default => generated 87 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 04:21:35.364328 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15137f4b-f2f9-4ce8-810e-88aa7ee5718d None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:21:35.365265 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a2ed932-73d9-47a0-90b5-08e22355235d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:21:35.365904 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a2ed932-73d9-47a0-90b5-08e22355235d None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:35.366185 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a2ed932-73d9-47a0-90b5-08e22355235d None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:35.366457 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a2ed932-73d9-47a0-90b5-08e22355235d None admin] PATH_INFO: `/v3/domains/default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:35.367578 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a2ed932-73d9-47a0-90b5-08e22355235d None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:21:35.369146 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a2ed932-73d9-47a0-90b5-08e22355235d None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:21:35.370722 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 10/19] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:21:35 2022] GET /identity/v3/domains/default => generated 199 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:21:35.376150 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a2ed932-73d9-47a0-90b5-08e22355235d None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:21:35.420444 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:21:35.456524 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:21:35.467501 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15137f4b-f2f9-4ce8-810e-88aa7ee5718d None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:21:35.475944 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a2ed932-73d9-47a0-90b5-08e22355235d None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:21:35.505488 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fdf1be3-c81f-4161-99a7-818e0103d1c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:21:35.506102 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fdf1be3-c81f-4161-99a7-818e0103d1c5 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:35.506382 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fdf1be3-c81f-4161-99a7-818e0103d1c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:35.506657 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fdf1be3-c81f-4161-99a7-818e0103d1c5 None admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:35.507304 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fdf1be3-c81f-4161-99a7-818e0103d1c5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:21:35.507609 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fdf1be3-c81f-4161-99a7-818e0103d1c5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:21:35.509022 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fdf1be3-c81f-4161-99a7-818e0103d1c5 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:21:35.510454 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36897d89-cda3-4cba-973a-408812229d37 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:21:35.511066 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36897d89-cda3-4cba-973a-408812229d37 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:35.511366 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36897d89-cda3-4cba-973a-408812229d37 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:35.511640 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36897d89-cda3-4cba-973a-408812229d37 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:35.512357 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36897d89-cda3-4cba-973a-408812229d37 None admin] RBAC: Adding query filter params (role.id=c51fe7c5ad804040879acfb54269455f, scope.domain.id=default, user.id=e61aa4e8d68f455e96668bc70e0a6679) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:21:35.512689 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36897d89-cda3-4cba-973a-408812229d37 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:21:35.514114 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36897d89-cda3-4cba-973a-408812229d37 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:21:35.521894 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 11/20] 10.222.0.25 () {56 vars in 1322 bytes} [Sun Sep 25 04:21:35 2022] GET /identity/v3/role_assignments?user.id=e61aa4e8d68f455e96668bc70e0a6679&scope.domain.id=default&role.id=c51fe7c5ad804040879acfb54269455f => generated 233 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:21:35.526597 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 10/21] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Sep 25 04:21:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:21:37.940991 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42577575-5b89-42a8-9bcc-273937d17607 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:37.941697 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42577575-5b89-42a8-9bcc-273937d17607 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:37.941697 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42577575-5b89-42a8-9bcc-273937d17607 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:37.942876 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 12/22] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:21:37 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:21:37.959265 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9eec2d0-9ef4-4e86-95a4-f453efb13e11 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:37.959771 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9eec2d0-9ef4-4e86-95a4-f453efb13e11 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:37.959873 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9eec2d0-9ef4-4e86-95a4-f453efb13e11 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:37.985040 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-a9eec2d0-9ef4-4e86-95a4-f453efb13e11 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:21:37.986268 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-a9eec2d0-9ef4-4e86-95a4-f453efb13e11 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:21:38.075962 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 11/23] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:21:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 04:21:38.217062 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67d9dfa4-aec1-41a8-8af4-e4c8ccd76825 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:38.217314 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67d9dfa4-aec1-41a8-8af4-e4c8ccd76825 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:38.217541 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67d9dfa4-aec1-41a8-8af4-e4c8ccd76825 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:38.218862 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 13/24] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:21:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:21:38.234433 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84613cb5-bbd9-440a-9408-dde312f2069b None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:38.234728 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84613cb5-bbd9-440a-9408-dde312f2069b None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:38.235008 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84613cb5-bbd9-440a-9408-dde312f2069b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:38.265653 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-84613cb5-bbd9-440a-9408-dde312f2069b None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:21:38.266853 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-84613cb5-bbd9-440a-9408-dde312f2069b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:21:38.319685 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:21:38.355129 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 12/25] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:21:38 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 04:21:38.515374 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67d9dfa4-aec1-41a8-8af4-e4c8ccd76825 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:21:38.651506 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67d9dfa4-aec1-41a8-8af4-e4c8ccd76825 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:21:38.653045 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a32a1c1c-3a68-43a5-8927-fddf7de16255 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:38.653598 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a32a1c1c-3a68-43a5-8927-fddf7de16255 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:38.654069 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a32a1c1c-3a68-43a5-8927-fddf7de16255 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:38.655616 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 13/26] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:21:38 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 25 04:21:38.665967 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a51ef95f-7501-4363-b80f-4773fc5bebe1 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:38.666551 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a51ef95f-7501-4363-b80f-4773fc5bebe1 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:38.667027 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a51ef95f-7501-4363-b80f-4773fc5bebe1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:38.698963 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37847213-b616-4ce1-bb48-986ae175911e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:21:38.699608 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37847213-b616-4ce1-bb48-986ae175911e None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:38.699891 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37847213-b616-4ce1-bb48-986ae175911e None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:38.700418 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37847213-b616-4ce1-bb48-986ae175911e None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:38.713654 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-a51ef95f-7501-4363-b80f-4773fc5bebe1 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:21:38.715173 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-a51ef95f-7501-4363-b80f-4773fc5bebe1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:21:38.716951 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37847213-b616-4ce1-bb48-986ae175911e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 04:21:38.724091 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-37847213-b616-4ce1-bb48-986ae175911e None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 04:21:38.724091 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:21:38.724091 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:21:38.724091 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:21:38.724091 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:21:38.724091 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:21:38.724091 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:21:38.724091 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:21:38.724091 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:21:38.724091 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:21:38.724091 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:21:38.724091 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:21:38.724091 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 04:21:38.724091 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 04:21:38.724091 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 04:21:38.724091 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 04:21:38.725029 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 04:21:38.725029 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 04:21:38.725029 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 04:21:38.725029 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 04:21:38.725029 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:21:38.725029 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:21:38.725029 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:21:38.725029 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:21:38.725029 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:21:38.725029 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:21:38.725029 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:21:38.725029 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:21:38.725029 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:21:38.725029 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:21:38.725029 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:21:38.725029 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:21:38.725806 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:21:38.725806 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:21:38.725806 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:21:38.725806 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:21:38.725806 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 04:21:38.725806 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 04:21:38.725806 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 04:21:38.725806 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 04:21:38.725806 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 04:21:38.725806 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 04:21:38.725806 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 04:21:38.725806 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:21:38.731268 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 14/27] 10.222.0.25 () {56 vars in 1096 bytes} [Sun Sep 25 04:21:38 2022] GET /identity/v3/roles/admin => generated 83 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:21:38.732023 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37847213-b616-4ce1-bb48-986ae175911e None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:21:38.742280 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:21:38.801680 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 14/28] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:21:38 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 25 04:21:38.803161 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f20b8a4-3efa-4f6a-8ac0-96bb0f654b5c None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:38.803642 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f20b8a4-3efa-4f6a-8ac0-96bb0f654b5c None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:38.804051 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f20b8a4-3efa-4f6a-8ac0-96bb0f654b5c None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:38.805583 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 15/29] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:21:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:21:38.806473 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f20b8a4-3efa-4f6a-8ac0-96bb0f654b5c None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:21:38.857823 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37847213-b616-4ce1-bb48-986ae175911e None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:21:38.897175 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dd0a596-1d76-4916-9ecb-4d2cb123a5bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:21:38.898346 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dd0a596-1d76-4916-9ecb-4d2cb123a5bc None admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:38.898794 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dd0a596-1d76-4916-9ecb-4d2cb123a5bc None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:38.899468 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dd0a596-1d76-4916-9ecb-4d2cb123a5bc None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:38.901236 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dd0a596-1d76-4916-9ecb-4d2cb123a5bc None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:21:38.902853 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dd0a596-1d76-4916-9ecb-4d2cb123a5bc None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:21:38.935045 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f20b8a4-3efa-4f6a-8ac0-96bb0f654b5c None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:21:38.945293 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:21:38.977450 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75febb32-22a9-43f0-9a07-b7f5583d9cbe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:21:38.978989 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75febb32-22a9-43f0-9a07-b7f5583d9cbe None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:38.979479 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75febb32-22a9-43f0-9a07-b7f5583d9cbe None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:38.979897 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75febb32-22a9-43f0-9a07-b7f5583d9cbe None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:38.980791 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75febb32-22a9-43f0-9a07-b7f5583d9cbe None admin] RBAC: Adding query filter params (name=admin) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:21:38.981323 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75febb32-22a9-43f0-9a07-b7f5583d9cbe None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:21:38.983057 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75febb32-22a9-43f0-9a07-b7f5583d9cbe None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:21:38.996212 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 16/30] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Sep 25 04:21:38 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:21:38.997811 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-747fc835-f198-494c-9add-16ebaa39c79a None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:38.998293 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-747fc835-f198-494c-9add-16ebaa39c79a None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:38.998744 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-747fc835-f198-494c-9add-16ebaa39c79a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:39.017638 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-747fc835-f198-494c-9add-16ebaa39c79a None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:21:39.019628 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-747fc835-f198-494c-9add-16ebaa39c79a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:21:39.056552 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 15/31] 10.222.0.25 () {60 vars in 1131 bytes} [Sun Sep 25 04:21:38 2022] POST /identity/v3/roles => generated 219 bytes in 325 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:21:39.062907 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dd0a596-1d76-4916-9ecb-4d2cb123a5bc None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:21:39.135501 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 17/32] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:21:38 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 04:21:39.186622 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:21:39.253923 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dd0a596-1d76-4916-9ecb-4d2cb123a5bc None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:21:39.296191 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76ca77d9-2c78-4561-a7b9-a6271754bc76 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:21:39.296873 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76ca77d9-2c78-4561-a7b9-a6271754bc76 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:39.297160 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76ca77d9-2c78-4561-a7b9-a6271754bc76 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:39.297438 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76ca77d9-2c78-4561-a7b9-a6271754bc76 None admin] PATH_INFO: `/v3/users/e61aa4e8d68f455e96668bc70e0a6679` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:39.298687 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76ca77d9-2c78-4561-a7b9-a6271754bc76 None admin] RBAC: Authorizing `identity:get_user(user_id=e61aa4e8d68f455e96668bc70e0a6679)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:21:39.299321 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-747fc835-f198-494c-9add-16ebaa39c79a None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:21:39.300787 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76ca77d9-2c78-4561-a7b9-a6271754bc76 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:21:39.302520 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 16/33] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Sep 25 04:21:39 2022] GET /identity/v3/users/e61aa4e8d68f455e96668bc70e0a6679 => generated 247 bytes in 245 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:21:39.307821 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76ca77d9-2c78-4561-a7b9-a6271754bc76 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:21:39.467018 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76ca77d9-2c78-4561-a7b9-a6271754bc76 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:21:39.504480 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-747fc835-f198-494c-9add-16ebaa39c79a None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:21:39.506113 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:21:39.527238 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:21:39.540175 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88b0b2b2-dbf3-4626-8843-fc706449826f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:21:39.541460 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88b0b2b2-dbf3-4626-8843-fc706449826f None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:39.541884 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88b0b2b2-dbf3-4626-8843-fc706449826f None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:39.542319 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88b0b2b2-dbf3-4626-8843-fc706449826f None admin] PATH_INFO: `/v3/domains/default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:39.542436 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6dd486e-ad08-4ee8-ab58-dc6e6476b70c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:21:39.543005 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6dd486e-ad08-4ee8-ab58-dc6e6476b70c None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:39.543298 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6dd486e-ad08-4ee8-ab58-dc6e6476b70c None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:39.543644 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6dd486e-ad08-4ee8-ab58-dc6e6476b70c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:39.544184 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88b0b2b2-dbf3-4626-8843-fc706449826f None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:21:39.546167 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88b0b2b2-dbf3-4626-8843-fc706449826f None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:21:39.548296 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 17/34] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:21:39 2022] GET /identity/v3/domains/default => generated 199 bytes in 241 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:21:39.552703 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88b0b2b2-dbf3-4626-8843-fc706449826f None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:21:39.560357 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6dd486e-ad08-4ee8-ab58-dc6e6476b70c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:21:39.563056 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6dd486e-ad08-4ee8-ab58-dc6e6476b70c None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:21:39.578044 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-f6dd486e-ad08-4ee8-ab58-dc6e6476b70c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:21:39.578044 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:21:39.578044 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:21:39.578044 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:21:39.578044 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:21:39.578044 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:21:39.578044 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:21:39.578044 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:21:39.578044 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:21:39.578044 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:21:39.578044 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:21:39.578044 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:21:39.578044 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 04:21:39.578044 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 04:21:39.578044 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 04:21:39.578044 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 04:21:39.579145 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:21:39.579145 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:21:39.579145 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:21:39.579145 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:21:39.579145 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:21:39.579145 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:21:39.579145 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:21:39.579145 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:21:39.579145 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:21:39.579145 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:21:39.579145 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:21:39.579145 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:21:39.579145 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:21:39.579145 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:21:39.579145 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:21:39.579145 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:21:39.579145 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 04:21:39.580123 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 04:21:39.580123 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:21:39.580123 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:21:39.580123 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 18/35] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:21:39 2022] GET /identity/v3/domains/Default => generated 87 bytes in 281 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:21:39.580123 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6dd486e-ad08-4ee8-ab58-dc6e6476b70c None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:21:39.676094 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88b0b2b2-dbf3-4626-8843-fc706449826f None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:21:39.689526 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6dd486e-ad08-4ee8-ab58-dc6e6476b70c None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:21:39.713444 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-308a0618-354d-4b74-9e4c-d621e675b67f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:21:39.714198 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-308a0618-354d-4b74-9e4c-d621e675b67f None admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:39.714586 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-308a0618-354d-4b74-9e4c-d621e675b67f None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:39.715063 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-308a0618-354d-4b74-9e4c-d621e675b67f None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:39.715959 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-308a0618-354d-4b74-9e4c-d621e675b67f None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:21:39.717486 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-308a0618-354d-4b74-9e4c-d621e675b67f None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:21:39.727180 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-595ff06f-44c7-4969-9180-a22f4b410bcd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:21:39.727854 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-595ff06f-44c7-4969-9180-a22f4b410bcd None admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:39.728257 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-595ff06f-44c7-4969-9180-a22f4b410bcd None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:39.728532 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-595ff06f-44c7-4969-9180-a22f4b410bcd None admin] PATH_INFO: `/v3/domains/default/users/e61aa4e8d68f455e96668bc70e0a6679/roles/c51fe7c5ad804040879acfb54269455f` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:39.731280 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-595ff06f-44c7-4969-9180-a22f4b410bcd None admin] RBAC: Authorizing `identity:create_grant(domain_id=default, user_id=e61aa4e8d68f455e96668bc70e0a6679, role_id=c51fe7c5ad804040879acfb54269455f)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:21:39.732747 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-595ff06f-44c7-4969-9180-a22f4b410bcd None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:21:39.735472 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 18/36] 10.222.0.25 () {60 vars in 1131 bytes} [Sun Sep 25 04:21:39 2022] POST /identity/v3/roles => generated 225 bytes in 186 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:21:39.736539 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-308a0618-354d-4b74-9e4c-d621e675b67f None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:21:39.865184 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:21:39.941674 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-308a0618-354d-4b74-9e4c-d621e675b67f None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:21:39.971941 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 19/37] 10.222.0.25 () {58 vars in 1361 bytes} [Sun Sep 25 04:21:39 2022] PUT /identity/v3/domains/default/users/e61aa4e8d68f455e96668bc70e0a6679/roles/c51fe7c5ad804040879acfb54269455f => generated 0 bytes in 392 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:21:39.974005 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a1986b-15b2-4b9f-bb61-e87184f5dff5 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:39.974303 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a1986b-15b2-4b9f-bb61-e87184f5dff5 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:39.974585 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a1986b-15b2-4b9f-bb61-e87184f5dff5 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:39.979663 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 20/38] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:21:39 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:21:39.985638 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d35ed7f-f08f-43ca-9b82-401d5afa8ce2 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:39.986022 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d35ed7f-f08f-43ca-9b82-401d5afa8ce2 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:39.986306 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d35ed7f-f08f-43ca-9b82-401d5afa8ce2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:39.988155 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04b6f55f-55aa-4ea2-b914-ea8427c6ff0b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:21:39.988831 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04b6f55f-55aa-4ea2-b914-ea8427c6ff0b None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:39.989118 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04b6f55f-55aa-4ea2-b914-ea8427c6ff0b None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:39.989398 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04b6f55f-55aa-4ea2-b914-ea8427c6ff0b None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:39.990065 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04b6f55f-55aa-4ea2-b914-ea8427c6ff0b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:21:39.990372 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04b6f55f-55aa-4ea2-b914-ea8427c6ff0b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:21:39.991821 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04b6f55f-55aa-4ea2-b914-ea8427c6ff0b None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:21:40.008215 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-4d35ed7f-f08f-43ca-9b82-401d5afa8ce2 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:21:40.009883 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-4d35ed7f-f08f-43ca-9b82-401d5afa8ce2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:21:40.013303 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:21:40.013832 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 19/39] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Sep 25 04:21:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 277 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:21:40.019729 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a197119c-5e6d-4cfd-86fb-e2fca73c0062 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:40.020023 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a197119c-5e6d-4cfd-86fb-e2fca73c0062 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:40.020305 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a197119c-5e6d-4cfd-86fb-e2fca73c0062 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:40.022407 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 20/40] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:21:40 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:21:40.023197 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a197119c-5e6d-4cfd-86fb-e2fca73c0062 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:21:40.095903 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 21/41] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:21:39 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 25 04:21:40.098162 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62fe2e16-0919-4ed0-b781-4d8f005fdea6 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:40.098414 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62fe2e16-0919-4ed0-b781-4d8f005fdea6 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:40.098692 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62fe2e16-0919-4ed0-b781-4d8f005fdea6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:40.119898 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-62fe2e16-0919-4ed0-b781-4d8f005fdea6 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:21:40.121095 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-62fe2e16-0919-4ed0-b781-4d8f005fdea6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:21:40.178465 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a197119c-5e6d-4cfd-86fb-e2fca73c0062 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:21:40.205776 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:21:40.207663 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 22/42] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:21:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 04:21:40.223194 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1fae423-411d-47d6-9cf5-1ad758e1c64b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:21:40.223820 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1fae423-411d-47d6-9cf5-1ad758e1c64b None admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:40.224104 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1fae423-411d-47d6-9cf5-1ad758e1c64b None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:40.224382 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1fae423-411d-47d6-9cf5-1ad758e1c64b None admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:40.225178 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1fae423-411d-47d6-9cf5-1ad758e1c64b None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:21:40.226715 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1fae423-411d-47d6-9cf5-1ad758e1c64b None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:21:40.262570 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 21/43] 10.222.0.25 () {60 vars in 1141 bytes} [Sun Sep 25 04:21:40 2022] POST /identity/v3/projects => generated 301 bytes in 239 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:21:40.385050 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62fe2e16-0919-4ed0-b781-4d8f005fdea6 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:21:40.461389 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:21:40.494685 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1fae423-411d-47d6-9cf5-1ad758e1c64b None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:21:40.507502 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62fe2e16-0919-4ed0-b781-4d8f005fdea6 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:21:40.547443 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65ecaf70-808a-4e40-97ce-8e05e41f2e27 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:21:40.548321 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ecaf70-808a-4e40-97ce-8e05e41f2e27 None admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:40.548601 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ecaf70-808a-4e40-97ce-8e05e41f2e27 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:40.548904 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ecaf70-808a-4e40-97ce-8e05e41f2e27 None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:40.550194 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65ecaf70-808a-4e40-97ce-8e05e41f2e27 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:21:40.551915 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65ecaf70-808a-4e40-97ce-8e05e41f2e27 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:21:40.572247 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 23/44] 10.222.0.25 () {60 vars in 1131 bytes} [Sun Sep 25 04:21:40 2022] POST /identity/v3/roles => generated 223 bytes in 188 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:21:40.731289 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1fae423-411d-47d6-9cf5-1ad758e1c64b None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:21:40.771255 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95a1d5a8-a2b4-430c-819c-7513cf9c1ec3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:21:40.771916 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a1d5a8-a2b4-430c-819c-7513cf9c1ec3 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:40.772190 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a1d5a8-a2b4-430c-819c-7513cf9c1ec3 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:40.772468 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a1d5a8-a2b4-430c-819c-7513cf9c1ec3 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:40.773616 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95a1d5a8-a2b4-430c-819c-7513cf9c1ec3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:21:40.775127 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95a1d5a8-a2b4-430c-819c-7513cf9c1ec3 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:21:40.776878 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 22/45] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:21:40 2022] GET /identity/v3/domains/default => generated 199 bytes in 283 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:21:40.782402 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65ecaf70-808a-4e40-97ce-8e05e41f2e27 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:21:40.885954 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65ecaf70-808a-4e40-97ce-8e05e41f2e27 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:21:40.913861 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:21:40.921081 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30724cd8-3444-41ea-ade6-b9de29cebd4d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:21:40.921930 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30724cd8-3444-41ea-ade6-b9de29cebd4d None admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:40.922426 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30724cd8-3444-41ea-ade6-b9de29cebd4d None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:40.922704 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30724cd8-3444-41ea-ade6-b9de29cebd4d None admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:40.923378 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30724cd8-3444-41ea-ade6-b9de29cebd4d None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:21:40.925861 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30724cd8-3444-41ea-ade6-b9de29cebd4d None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:21:40.955640 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 24/46] 10.222.0.25 () {60 vars in 1141 bytes} [Sun Sep 25 04:21:40 2022] POST /identity/v3/projects => generated 312 bytes in 174 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:21:42.208363 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-350f8983-1c1a-4ff6-ba82-73163951908e None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:42.208926 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-350f8983-1c1a-4ff6-ba82-73163951908e None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:42.209036 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-350f8983-1c1a-4ff6-ba82-73163951908e None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:42.210305 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 23/47] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:21:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:21:42.215623 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6beded19-e927-4c5c-b14e-08c5862e2858 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:42.216016 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6beded19-e927-4c5c-b14e-08c5862e2858 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:42.216186 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6beded19-e927-4c5c-b14e-08c5862e2858 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:42.234555 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-6beded19-e927-4c5c-b14e-08c5862e2858 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:21:42.235746 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-6beded19-e927-4c5c-b14e-08c5862e2858 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:21:42.301725 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 25/48] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:21:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 04:21:42.569924 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-350f8983-1c1a-4ff6-ba82-73163951908e None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:21:42.615229 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:21:42.676888 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-350f8983-1c1a-4ff6-ba82-73163951908e None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:21:42.710363 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5c9f062-8acc-4ac8-8def-38102ba3b1e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:21:42.710996 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5c9f062-8acc-4ac8-8def-38102ba3b1e8 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:42.711278 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5c9f062-8acc-4ac8-8def-38102ba3b1e8 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:42.711550 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5c9f062-8acc-4ac8-8def-38102ba3b1e8 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:42.722067 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5c9f062-8acc-4ac8-8def-38102ba3b1e8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 04:21:42.726878 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-c5c9f062-8acc-4ac8-8def-38102ba3b1e8 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 04:21:42.726878 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:21:42.726878 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:21:42.726878 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:21:42.726878 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:21:42.726878 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:21:42.726878 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:21:42.726878 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:21:42.726878 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:21:42.726878 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:21:42.726878 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:21:42.726878 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:21:42.726878 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 04:21:42.726878 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 04:21:42.726878 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 04:21:42.726878 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 04:21:42.727896 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 04:21:42.727896 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 04:21:42.727896 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 04:21:42.727896 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 04:21:42.727896 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:21:42.727896 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:21:42.727896 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:21:42.727896 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:21:42.727896 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:21:42.727896 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:21:42.727896 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:21:42.727896 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:21:42.727896 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:21:42.727896 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:21:42.727896 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:21:42.727896 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:21:42.729110 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:21:42.729110 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:21:42.729110 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:21:42.729110 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:21:42.729110 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 04:21:42.729110 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 04:21:42.729110 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 04:21:42.729110 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 04:21:42.729110 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 04:21:42.729110 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 04:21:42.729110 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 04:21:42.729110 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:21:42.729110 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 24/49] 10.222.0.25 () {56 vars in 1096 bytes} [Sun Sep 25 04:21:42 2022] GET /identity/v3/roles/admin => generated 83 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:21:42.733743 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6beded19-e927-4c5c-b14e-08c5862e2858 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:21:42.831089 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6beded19-e927-4c5c-b14e-08c5862e2858 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:21:42.843825 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:21:42.865817 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8ecf320-c3ed-474a-8598-663bdaa7dbe8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:21:42.866431 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8ecf320-c3ed-474a-8598-663bdaa7dbe8 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:42.866713 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8ecf320-c3ed-474a-8598-663bdaa7dbe8 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:42.866988 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8ecf320-c3ed-474a-8598-663bdaa7dbe8 None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:42.867654 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8ecf320-c3ed-474a-8598-663bdaa7dbe8 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:21:42.867961 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8ecf320-c3ed-474a-8598-663bdaa7dbe8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:21:42.869377 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8ecf320-c3ed-474a-8598-663bdaa7dbe8 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:21:42.877518 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 26/50] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Sep 25 04:21:42 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:21:42.883078 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5c9f062-8acc-4ac8-8def-38102ba3b1e8 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:21:42.979026 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:21:42.982596 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5c9f062-8acc-4ac8-8def-38102ba3b1e8 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:21:43.021275 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-791d10e4-615e-4569-a44f-e48cff283991 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:21:43.021936 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-791d10e4-615e-4569-a44f-e48cff283991 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:43.022226 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-791d10e4-615e-4569-a44f-e48cff283991 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:43.022509 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-791d10e4-615e-4569-a44f-e48cff283991 None admin] PATH_INFO: `/v3/users/e61aa4e8d68f455e96668bc70e0a6679` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:43.023722 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-791d10e4-615e-4569-a44f-e48cff283991 None admin] RBAC: Authorizing `identity:get_user(user_id=e61aa4e8d68f455e96668bc70e0a6679)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:21:43.025256 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-791d10e4-615e-4569-a44f-e48cff283991 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:21:43.026893 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 25/51] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Sep 25 04:21:42 2022] GET /identity/v3/users/e61aa4e8d68f455e96668bc70e0a6679 => generated 247 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:21:43.032394 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8ecf320-c3ed-474a-8598-663bdaa7dbe8 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:21:43.119939 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8df6dce-b8af-4f3f-8d29-42efb99b307d None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:43.120223 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8df6dce-b8af-4f3f-8d29-42efb99b307d None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:43.120504 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8df6dce-b8af-4f3f-8d29-42efb99b307d None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:43.121892 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 26/52] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:21:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 25 04:21:43.127771 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-369a4f02-6ca4-424c-a260-19e029349051 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:43.128063 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-369a4f02-6ca4-424c-a260-19e029349051 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:43.128336 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-369a4f02-6ca4-424c-a260-19e029349051 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:43.147956 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-369a4f02-6ca4-424c-a260-19e029349051 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:21:43.149606 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-369a4f02-6ca4-424c-a260-19e029349051 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:21:43.225164 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 27/53] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:21:43 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 25 04:21:43.242875 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8ecf320-c3ed-474a-8598-663bdaa7dbe8 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:21:43.276955 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-852e59bc-7e8f-40ad-bfc1-00504d22083a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:21:43.277587 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-852e59bc-7e8f-40ad-bfc1-00504d22083a None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:43.277875 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-852e59bc-7e8f-40ad-bfc1-00504d22083a None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:43.278151 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-852e59bc-7e8f-40ad-bfc1-00504d22083a None admin] PATH_INFO: `/v3/domains/default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:43.279278 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-852e59bc-7e8f-40ad-bfc1-00504d22083a None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:21:43.280736 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-852e59bc-7e8f-40ad-bfc1-00504d22083a None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:21:43.282307 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 27/54] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:21:43 2022] GET /identity/v3/domains/default => generated 199 bytes in 250 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:21:43.288154 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-369a4f02-6ca4-424c-a260-19e029349051 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:21:43.383883 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:21:43.388089 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-369a4f02-6ca4-424c-a260-19e029349051 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:21:43.421980 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c68d95c-3840-4e75-84aa-bb0bc75badba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:21:43.422584 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c68d95c-3840-4e75-84aa-bb0bc75badba None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:43.422867 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c68d95c-3840-4e75-84aa-bb0bc75badba None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:43.423143 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c68d95c-3840-4e75-84aa-bb0bc75badba None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:43.423859 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c68d95c-3840-4e75-84aa-bb0bc75badba None admin] RBAC: Adding query filter params (role.id=c51fe7c5ad804040879acfb54269455f, scope.domain.id=default, user.id=e61aa4e8d68f455e96668bc70e0a6679) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:21:43.424751 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c68d95c-3840-4e75-84aa-bb0bc75badba None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:21:43.426233 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c68d95c-3840-4e75-84aa-bb0bc75badba None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:21:43.432034 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 28/55] 10.222.0.25 () {56 vars in 1322 bytes} [Sun Sep 25 04:21:43 2022] GET /identity/v3/role_assignments?user.id=e61aa4e8d68f455e96668bc70e0a6679&scope.domain.id=default&role.id=c51fe7c5ad804040879acfb54269455f => generated 530 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:21:43.500019 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-852e59bc-7e8f-40ad-bfc1-00504d22083a None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:21:43.546110 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:21:43.606814 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-852e59bc-7e8f-40ad-bfc1-00504d22083a None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:21:43.640868 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ab93dae-4b1e-42a6-ba68-31dedd3a6ec1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:21:43.641510 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ab93dae-4b1e-42a6-ba68-31dedd3a6ec1 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:43.641792 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ab93dae-4b1e-42a6-ba68-31dedd3a6ec1 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:43.642068 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ab93dae-4b1e-42a6-ba68-31dedd3a6ec1 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:43.644004 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ab93dae-4b1e-42a6-ba68-31dedd3a6ec1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:21:43.645517 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ab93dae-4b1e-42a6-ba68-31dedd3a6ec1 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:21:43.647101 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 28/56] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:21:43 2022] GET /identity/v3/domains/default => generated 199 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:21:43.653108 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c68d95c-3840-4e75-84aa-bb0bc75badba None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:21:43.755709 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c68d95c-3840-4e75-84aa-bb0bc75badba None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:21:43.788771 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e2e7d8b-9c05-4619-b35e-9c69a9b17e3d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:21:43.789370 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e2e7d8b-9c05-4619-b35e-9c69a9b17e3d None admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:43.789649 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e2e7d8b-9c05-4619-b35e-9c69a9b17e3d None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:43.789922 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e2e7d8b-9c05-4619-b35e-9c69a9b17e3d None admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:43.790564 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e2e7d8b-9c05-4619-b35e-9c69a9b17e3d None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:21:43.791995 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e2e7d8b-9c05-4619-b35e-9c69a9b17e3d None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:21:43.818097 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 29/57] 10.222.0.25 () {60 vars in 1140 bytes} [Sun Sep 25 04:21:43 2022] POST /identity/v3/projects => generated 298 bytes in 165 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:21:45.953687 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc38dbf9-7afb-48a7-931c-125900f7e7ac None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:45.954194 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc38dbf9-7afb-48a7-931c-125900f7e7ac None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:45.954281 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc38dbf9-7afb-48a7-931c-125900f7e7ac None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:45.955709 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 29/58] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:21:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:21:45.960786 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ed1e50-0881-4946-abe8-da4d86c11327 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:45.961210 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ed1e50-0881-4946-abe8-da4d86c11327 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:45.961346 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ed1e50-0881-4946-abe8-da4d86c11327 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:45.979460 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-04ed1e50-0881-4946-abe8-da4d86c11327 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:21:45.980681 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-04ed1e50-0881-4946-abe8-da4d86c11327 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:21:45.990326 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:21:46.046517 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 30/59] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:21:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 04:21:46.319841 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc38dbf9-7afb-48a7-931c-125900f7e7ac None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:21:46.418423 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc38dbf9-7afb-48a7-931c-125900f7e7ac None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:21:46.428351 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:21:46.451071 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef390999-61c9-4bfd-98fb-00f3369588e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:21:46.451692 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef390999-61c9-4bfd-98fb-00f3369588e1 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:46.451973 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef390999-61c9-4bfd-98fb-00f3369588e1 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:46.452245 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef390999-61c9-4bfd-98fb-00f3369588e1 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:46.453385 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef390999-61c9-4bfd-98fb-00f3369588e1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:21:46.454823 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef390999-61c9-4bfd-98fb-00f3369588e1 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:21:46.456961 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 30/60] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:21:46 2022] GET /identity/v3/domains/default => generated 199 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:21:46.462413 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04ed1e50-0881-4946-abe8-da4d86c11327 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:21:46.558474 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04ed1e50-0881-4946-abe8-da4d86c11327 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:21:46.591521 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b95dee01-6922-44e8-9df0-a0d32b6568ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:21:46.592066 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b95dee01-6922-44e8-9df0-a0d32b6568ea None admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:46.592348 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b95dee01-6922-44e8-9df0-a0d32b6568ea None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:46.592623 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b95dee01-6922-44e8-9df0-a0d32b6568ea None admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:46.593298 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b95dee01-6922-44e8-9df0-a0d32b6568ea None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:21:46.594740 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b95dee01-6922-44e8-9df0-a0d32b6568ea None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:21:46.618005 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:21:46.622505 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 31/61] 10.222.0.25 () {60 vars in 1132 bytes} [Sun Sep 25 04:21:46 2022] POST /identity/v3/users => generated 275 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:21:50.011920 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f3744c6-8939-4c7a-b333-a80926216676 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:50.011920 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f3744c6-8939-4c7a-b333-a80926216676 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:50.011920 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f3744c6-8939-4c7a-b333-a80926216676 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:50.011920 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 31/62] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:21:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:21:50.017756 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e981a2-735b-4a04-b34f-db8ef017d80f None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:50.018216 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e981a2-735b-4a04-b34f-db8ef017d80f None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:50.018616 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e981a2-735b-4a04-b34f-db8ef017d80f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:50.045765 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-e0e981a2-735b-4a04-b34f-db8ef017d80f None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:21:50.047003 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-e0e981a2-735b-4a04-b34f-db8ef017d80f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:21:50.141959 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3921c22-621d-45fd-a0dc-5274d62103a1 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:50.142299 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 32/63] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:21:50 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 04:21:50.143302 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3921c22-621d-45fd-a0dc-5274d62103a1 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:50.143783 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3921c22-621d-45fd-a0dc-5274d62103a1 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:50.145436 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 32/64] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:21:50 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:21:50.165829 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fee8c64-4492-4b02-808f-44f84d431440 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:50.166111 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fee8c64-4492-4b02-808f-44f84d431440 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:50.166389 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fee8c64-4492-4b02-808f-44f84d431440 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:50.192383 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-2fee8c64-4492-4b02-808f-44f84d431440 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:21:50.193627 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-2fee8c64-4492-4b02-808f-44f84d431440 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:21:50.279289 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 33/65] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:21:50 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 04:21:50.484951 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc9a21c9-691d-496f-8ad7-a8b9903823c0 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:50.485605 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc9a21c9-691d-496f-8ad7-a8b9903823c0 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:50.486126 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc9a21c9-691d-496f-8ad7-a8b9903823c0 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:50.488117 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 33/66] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:21:50 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:21:50.494832 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffe4b955-ec41-4d65-904a-2bc57a9aff96 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:50.495474 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffe4b955-ec41-4d65-904a-2bc57a9aff96 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:50.495952 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffe4b955-ec41-4d65-904a-2bc57a9aff96 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:50.517228 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-ffe4b955-ec41-4d65-904a-2bc57a9aff96 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:21:50.518911 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-ffe4b955-ec41-4d65-904a-2bc57a9aff96 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:21:50.582417 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:21:50.595344 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 34/67] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:21:50 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 04:21:50.601307 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc9a21c9-691d-496f-8ad7-a8b9903823c0 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:21:50.627673 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a903027f-d3ff-44ef-9925-275863a57ca7 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:50.628463 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a903027f-d3ff-44ef-9925-275863a57ca7 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:50.628963 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a903027f-d3ff-44ef-9925-275863a57ca7 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:50.630603 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 35/68] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:21:50 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:21:50.631545 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a903027f-d3ff-44ef-9925-275863a57ca7 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:21:50.731197 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc9a21c9-691d-496f-8ad7-a8b9903823c0 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:21:50.750314 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a903027f-d3ff-44ef-9925-275863a57ca7 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:21:50.765877 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d2e253c-f837-456e-acee-7e0e3492ad95 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:21:50.766513 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d2e253c-f837-456e-acee-7e0e3492ad95 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:50.766800 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d2e253c-f837-456e-acee-7e0e3492ad95 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:50.767077 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d2e253c-f837-456e-acee-7e0e3492ad95 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:50.784200 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d2e253c-f837-456e-acee-7e0e3492ad95 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 04:21:50.786929 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f073190a-0979-4706-90c8-d98a81807efb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:21:50.787570 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f073190a-0979-4706-90c8-d98a81807efb None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:50.787858 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f073190a-0979-4706-90c8-d98a81807efb None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:50.788138 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f073190a-0979-4706-90c8-d98a81807efb None admin] PATH_INFO: `/v3/roles/member` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:50.790454 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-3d2e253c-f837-456e-acee-7e0e3492ad95 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 04:21:50.790454 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:21:50.790454 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:21:50.790454 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:21:50.790454 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:21:50.790454 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:21:50.790454 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:21:50.790454 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:21:50.790454 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:21:50.790454 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:21:50.790454 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:21:50.790454 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:21:50.790454 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 04:21:50.790454 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 04:21:50.790454 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 04:21:50.790454 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 04:21:50.791281 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 04:21:50.791281 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 04:21:50.791281 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 04:21:50.791281 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 04:21:50.791281 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:21:50.791281 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:21:50.791281 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:21:50.791281 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:21:50.791281 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:21:50.791281 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:21:50.791281 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:21:50.791281 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:21:50.791281 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:21:50.791281 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:21:50.791281 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:21:50.791281 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:21:50.792052 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:21:50.792052 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:21:50.792052 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:21:50.792052 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:21:50.792052 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 04:21:50.792052 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 04:21:50.792052 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 04:21:50.792052 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 04:21:50.792052 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 04:21:50.792052 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 04:21:50.792052 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 04:21:50.792052 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:21:50.792052 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 34/69] 10.222.0.25 () {56 vars in 1096 bytes} [Sun Sep 25 04:21:50 2022] GET /identity/v3/roles/admin => generated 83 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:21:50.792724 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-485980b8-6244-4a04-80e2-99f8548133a9 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:50.792996 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-485980b8-6244-4a04-80e2-99f8548133a9 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:50.793409 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-485980b8-6244-4a04-80e2-99f8548133a9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:50.804257 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f073190a-0979-4706-90c8-d98a81807efb None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 04:21:50.809223 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-f073190a-0979-4706-90c8-d98a81807efb None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 25 04:21:50.809223 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:21:50.809223 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:21:50.809223 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:21:50.809223 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:21:50.809223 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:21:50.809223 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:21:50.809223 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:21:50.809223 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:21:50.809223 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:21:50.809223 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:21:50.809223 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:21:50.809223 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 04:21:50.809223 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 04:21:50.809223 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 04:21:50.809223 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 04:21:50.810064 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 04:21:50.810064 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 04:21:50.810064 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 04:21:50.810064 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 04:21:50.810064 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:21:50.810064 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:21:50.810064 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:21:50.810064 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:21:50.810064 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:21:50.810064 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:21:50.810064 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:21:50.810064 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:21:50.810064 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:21:50.810064 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:21:50.810064 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:21:50.810064 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:21:50.810832 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:21:50.810832 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:21:50.810832 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:21:50.810832 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:21:50.810832 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 04:21:50.810832 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 04:21:50.810832 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 04:21:50.810832 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 04:21:50.810832 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 04:21:50.810832 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 04:21:50.810832 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 25 04:21:50.810832 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:21:50.810832 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 36/70] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Sep 25 04:21:50 2022] GET /identity/v3/roles/member => generated 84 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:21:50.812495 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d509aa8-3d22-48b0-91be-fb4049373d99 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:50.812495 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d509aa8-3d22-48b0-91be-fb4049373d99 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:50.812495 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d509aa8-3d22-48b0-91be-fb4049373d99 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:50.813445 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-485980b8-6244-4a04-80e2-99f8548133a9 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:21:50.813737 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 37/71] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:21:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:21:50.815215 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-485980b8-6244-4a04-80e2-99f8548133a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:21:50.818745 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d509aa8-3d22-48b0-91be-fb4049373d99 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:21:50.847805 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:21:50.874926 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:21:50.891056 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 35/72] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:21:50 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 04:21:50.896940 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-485980b8-6244-4a04-80e2-99f8548133a9 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:21:50.935379 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d509aa8-3d22-48b0-91be-fb4049373d99 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:21:50.972387 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ddd2f74-48a4-45ab-996c-3f99a1a8d514 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:21:50.973033 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ddd2f74-48a4-45ab-996c-3f99a1a8d514 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:50.973323 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ddd2f74-48a4-45ab-996c-3f99a1a8d514 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:50.973603 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ddd2f74-48a4-45ab-996c-3f99a1a8d514 None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:50.974389 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ddd2f74-48a4-45ab-996c-3f99a1a8d514 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:21:50.974704 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ddd2f74-48a4-45ab-996c-3f99a1a8d514 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:21:50.976248 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ddd2f74-48a4-45ab-996c-3f99a1a8d514 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:21:50.985738 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 38/73] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Sep 25 04:21:50 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:21:50.992275 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-247139fb-ccf4-4534-8c73-1c605ddff168 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:50.992566 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-247139fb-ccf4-4534-8c73-1c605ddff168 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:50.993019 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-247139fb-ccf4-4534-8c73-1c605ddff168 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:50.994298 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-485980b8-6244-4a04-80e2-99f8548133a9 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:21:51.011078 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-247139fb-ccf4-4534-8c73-1c605ddff168 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:21:51.012460 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-247139fb-ccf4-4534-8c73-1c605ddff168 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:21:51.030511 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34f8af1d-a5c8-4972-975a-6e76059d2562 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:21:51.031130 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34f8af1d-a5c8-4972-975a-6e76059d2562 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:51.031414 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34f8af1d-a5c8-4972-975a-6e76059d2562 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:51.031688 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34f8af1d-a5c8-4972-975a-6e76059d2562 None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:51.032353 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34f8af1d-a5c8-4972-975a-6e76059d2562 None admin] RBAC: Adding query filter params (name=member) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:21:51.032687 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34f8af1d-a5c8-4972-975a-6e76059d2562 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:21:51.034154 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34f8af1d-a5c8-4972-975a-6e76059d2562 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:21:51.042124 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 36/74] 10.222.0.25 () {56 vars in 1101 bytes} [Sun Sep 25 04:21:50 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:21:51.042904 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34f8af1d-a5c8-4972-975a-6e76059d2562 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:21:51.077916 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:21:51.089015 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 39/75] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:21:50 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 04:21:51.097461 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-247139fb-ccf4-4534-8c73-1c605ddff168 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:21:51.147354 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:21:51.166170 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34f8af1d-a5c8-4972-975a-6e76059d2562 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:21:51.203197 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e5bec7c-5021-40a7-90da-550a97685be3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:21:51.203840 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e5bec7c-5021-40a7-90da-550a97685be3 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:51.204125 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e5bec7c-5021-40a7-90da-550a97685be3 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:51.204393 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e5bec7c-5021-40a7-90da-550a97685be3 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:51.211229 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-247139fb-ccf4-4534-8c73-1c605ddff168 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:21:51.214895 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e5bec7c-5021-40a7-90da-550a97685be3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 04:21:51.220169 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-2e5bec7c-5021-40a7-90da-550a97685be3 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 25 04:21:51.220169 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:21:51.220169 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:21:51.220169 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:21:51.220169 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:21:51.220169 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:21:51.220169 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:21:51.220169 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:21:51.220169 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:21:51.220169 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:21:51.220169 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:21:51.220169 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:21:51.220169 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 04:21:51.220169 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 04:21:51.220169 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 04:21:51.220169 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 04:21:51.221638 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 04:21:51.221638 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 04:21:51.221638 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 04:21:51.221638 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 04:21:51.221638 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:21:51.221638 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:21:51.221638 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:21:51.221638 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:21:51.221638 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:21:51.221638 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:21:51.221638 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:21:51.221638 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:21:51.221638 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:21:51.221638 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:21:51.221638 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:21:51.221638 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:21:51.222408 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:21:51.222408 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:21:51.222408 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:21:51.222408 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:21:51.222408 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 04:21:51.222408 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 04:21:51.222408 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 04:21:51.222408 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 04:21:51.222408 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 04:21:51.222408 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 04:21:51.222408 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 25 04:21:51.222408 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:21:51.222408 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 37/76] 10.222.0.25 () {56 vars in 1114 bytes} [Sun Sep 25 04:21:51 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:21:51.224873 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e5bec7c-5021-40a7-90da-550a97685be3 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:21:51.262442 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c369be3c-59ab-4d49-bb81-061727716561 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:21:51.263671 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c369be3c-59ab-4d49-bb81-061727716561 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:51.264218 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c369be3c-59ab-4d49-bb81-061727716561 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:51.264713 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c369be3c-59ab-4d49-bb81-061727716561 None admin] PATH_INFO: `/v3/users/e61aa4e8d68f455e96668bc70e0a6679` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:51.266506 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c369be3c-59ab-4d49-bb81-061727716561 None admin] RBAC: Authorizing `identity:get_user(user_id=e61aa4e8d68f455e96668bc70e0a6679)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:21:51.269891 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c369be3c-59ab-4d49-bb81-061727716561 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:21:51.272892 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 40/77] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Sep 25 04:21:51 2022] GET /identity/v3/users/e61aa4e8d68f455e96668bc70e0a6679 => generated 247 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:21:51.274028 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c369be3c-59ab-4d49-bb81-061727716561 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:21:51.343746 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:21:51.356781 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e5bec7c-5021-40a7-90da-550a97685be3 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:21:51.383450 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c369be3c-59ab-4d49-bb81-061727716561 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:21:51.391170 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cd42c49-80ad-48bc-99e6-4552ecc9cc9c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:21:51.391815 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cd42c49-80ad-48bc-99e6-4552ecc9cc9c None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:51.392096 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cd42c49-80ad-48bc-99e6-4552ecc9cc9c None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:51.392370 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cd42c49-80ad-48bc-99e6-4552ecc9cc9c None admin] PATH_INFO: `/v3/users/bc82e617c36e4fdfa13e159395d54e1c` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:51.395179 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:21:51.409055 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cd42c49-80ad-48bc-99e6-4552ecc9cc9c None admin] RBAC: Authorizing `identity:get_user(user_id=bc82e617c36e4fdfa13e159395d54e1c)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:21:51.410458 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cd42c49-80ad-48bc-99e6-4552ecc9cc9c None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:21:51.412095 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 38/78] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Sep 25 04:21:51 2022] GET /identity/v3/users/bc82e617c36e4fdfa13e159395d54e1c => generated 275 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:21:51.412886 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cd42c49-80ad-48bc-99e6-4552ecc9cc9c None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:21:51.422290 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d904fcbd-71b0-49c7-8e09-dfa9277604fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:21:51.422931 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d904fcbd-71b0-49c7-8e09-dfa9277604fb None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:51.423216 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d904fcbd-71b0-49c7-8e09-dfa9277604fb None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:51.423492 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d904fcbd-71b0-49c7-8e09-dfa9277604fb None admin] PATH_INFO: `/v3/roles/member` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:51.433966 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d904fcbd-71b0-49c7-8e09-dfa9277604fb None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 04:21:51.438981 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-d904fcbd-71b0-49c7-8e09-dfa9277604fb None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 25 04:21:51.438981 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:21:51.438981 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:21:51.438981 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:21:51.438981 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:21:51.438981 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:21:51.438981 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:21:51.438981 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:21:51.438981 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:21:51.438981 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:21:51.438981 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:21:51.438981 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:21:51.438981 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 04:21:51.438981 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 04:21:51.438981 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 04:21:51.438981 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 04:21:51.440402 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 04:21:51.440402 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 04:21:51.440402 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 04:21:51.440402 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 04:21:51.440402 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:21:51.440402 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:21:51.440402 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:21:51.440402 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:21:51.440402 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:21:51.440402 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:21:51.440402 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:21:51.440402 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:21:51.440402 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:21:51.440402 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:21:51.440402 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:21:51.440402 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:21:51.441250 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:21:51.441250 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:21:51.441250 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:21:51.441250 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:21:51.441250 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 04:21:51.441250 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 04:21:51.441250 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 04:21:51.441250 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 04:21:51.441250 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 04:21:51.441250 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 04:21:51.441250 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 25 04:21:51.441250 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:21:51.441250 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 41/79] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Sep 25 04:21:51 2022] GET /identity/v3/roles/member => generated 84 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:21:51.441889 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d904fcbd-71b0-49c7-8e09-dfa9277604fb None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:21:51.515152 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cd42c49-80ad-48bc-99e6-4552ecc9cc9c None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:21:51.540358 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d904fcbd-71b0-49c7-8e09-dfa9277604fb None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:21:51.549141 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41f646a1-2f0a-485c-b2e6-515747330f28 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:21:51.549762 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41f646a1-2f0a-485c-b2e6-515747330f28 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:51.550043 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41f646a1-2f0a-485c-b2e6-515747330f28 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:51.550321 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41f646a1-2f0a-485c-b2e6-515747330f28 None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:51.550986 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41f646a1-2f0a-485c-b2e6-515747330f28 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:21:51.551293 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41f646a1-2f0a-485c-b2e6-515747330f28 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:21:51.552775 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41f646a1-2f0a-485c-b2e6-515747330f28 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:21:51.560686 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 39/80] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Sep 25 04:21:51 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:21:51.561392 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41f646a1-2f0a-485c-b2e6-515747330f28 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:21:51.573093 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:21:51.577721 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa902995-6a68-451e-b35c-502ac565d9a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:21:51.578339 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa902995-6a68-451e-b35c-502ac565d9a2 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:51.578674 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa902995-6a68-451e-b35c-502ac565d9a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:51.578955 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa902995-6a68-451e-b35c-502ac565d9a2 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:51.619864 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:21:51.649205 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa902995-6a68-451e-b35c-502ac565d9a2 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:21:51.650688 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa902995-6a68-451e-b35c-502ac565d9a2 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:21:51.661045 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41f646a1-2f0a-485c-b2e6-515747330f28 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:21:51.694520 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75d69cdc-fb27-487f-85db-6d472dfbc311 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:21:51.695155 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d69cdc-fb27-487f-85db-6d472dfbc311 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:51.695439 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d69cdc-fb27-487f-85db-6d472dfbc311 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:51.695718 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d69cdc-fb27-487f-85db-6d472dfbc311 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:51.702049 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75d69cdc-fb27-487f-85db-6d472dfbc311 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:21:51.703544 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75d69cdc-fb27-487f-85db-6d472dfbc311 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:21:51.705397 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 40/81] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:21:51 2022] GET /identity/v3/domains/default => generated 199 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:21:51.705968 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75d69cdc-fb27-487f-85db-6d472dfbc311 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:21:51.752960 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 42/82] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Sep 25 04:21:51 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 312 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 25 04:21:51.757645 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa902995-6a68-451e-b35c-502ac565d9a2 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:21:51.784715 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:21:51.806603 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75d69cdc-fb27-487f-85db-6d472dfbc311 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:21:51.825541 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:21:51.839991 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01c50f70-b30f-4942-8c0e-4ea838439a9e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:21:51.840610 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01c50f70-b30f-4942-8c0e-4ea838439a9e None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:51.840922 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01c50f70-b30f-4942-8c0e-4ea838439a9e None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:51.841202 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01c50f70-b30f-4942-8c0e-4ea838439a9e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:51.856256 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa902995-6a68-451e-b35c-502ac565d9a2 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:21:51.889358 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-800be4a0-f2a1-4c77-9dbb-818c59eabbbc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:21:51.889966 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-800be4a0-f2a1-4c77-9dbb-818c59eabbbc None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:51.890244 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-800be4a0-f2a1-4c77-9dbb-818c59eabbbc None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:51.890518 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-800be4a0-f2a1-4c77-9dbb-818c59eabbbc None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:51.891174 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-800be4a0-f2a1-4c77-9dbb-818c59eabbbc None admin] RBAC: Adding query filter params (name=member) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:21:51.891476 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-800be4a0-f2a1-4c77-9dbb-818c59eabbbc None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:21:51.893624 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-800be4a0-f2a1-4c77-9dbb-818c59eabbbc None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:21:51.901750 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 43/83] 10.222.0.25 () {56 vars in 1101 bytes} [Sun Sep 25 04:21:51 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:21:51.902533 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-800be4a0-f2a1-4c77-9dbb-818c59eabbbc None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:21:51.914490 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01c50f70-b30f-4942-8c0e-4ea838439a9e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:21:51.915898 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01c50f70-b30f-4942-8c0e-4ea838439a9e None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:21:51.993865 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:21:52.000933 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-800be4a0-f2a1-4c77-9dbb-818c59eabbbc None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:21:52.016606 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 41/84] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Sep 25 04:21:51 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 25 04:21:52.017437 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01c50f70-b30f-4942-8c0e-4ea838439a9e None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:21:52.030990 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:21:52.036795 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b751f79-b939-487d-aa74-e2fc0d12925e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:21:52.037437 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b751f79-b939-487d-aa74-e2fc0d12925e None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:52.037721 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b751f79-b939-487d-aa74-e2fc0d12925e None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:52.037995 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b751f79-b939-487d-aa74-e2fc0d12925e None admin] PATH_INFO: `/v3/users/bc82e617c36e4fdfa13e159395d54e1c` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:52.039156 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b751f79-b939-487d-aa74-e2fc0d12925e None admin] RBAC: Authorizing `identity:get_user(user_id=bc82e617c36e4fdfa13e159395d54e1c)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:21:52.040599 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b751f79-b939-487d-aa74-e2fc0d12925e None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:21:52.042203 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 44/85] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Sep 25 04:21:51 2022] GET /identity/v3/users/bc82e617c36e4fdfa13e159395d54e1c => generated 275 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:21:52.042963 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b751f79-b939-487d-aa74-e2fc0d12925e None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:21:52.114973 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01c50f70-b30f-4942-8c0e-4ea838439a9e None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:21:52.145129 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b751f79-b939-487d-aa74-e2fc0d12925e None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:21:52.148733 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-992c252d-8805-4f24-bad9-5341c07bae19 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:21:52.149500 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-992c252d-8805-4f24-bad9-5341c07bae19 None admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:52.149990 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-992c252d-8805-4f24-bad9-5341c07bae19 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:52.150274 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-992c252d-8805-4f24-bad9-5341c07bae19 None admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:52.150936 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-992c252d-8805-4f24-bad9-5341c07bae19 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:21:52.152432 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-992c252d-8805-4f24-bad9-5341c07bae19 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:21:52.179640 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8435c20-f1e3-4930-b61b-65e7bc0f8cb6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:21:52.192856 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 42/86] 10.222.0.25 () {60 vars in 1132 bytes} [Sun Sep 25 04:21:52 2022] POST /identity/v3/users => generated 289 bytes in 163 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:21:52.193074 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8435c20-f1e3-4930-b61b-65e7bc0f8cb6 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:52.193074 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8435c20-f1e3-4930-b61b-65e7bc0f8cb6 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:52.193074 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8435c20-f1e3-4930-b61b-65e7bc0f8cb6 None admin] PATH_INFO: `/v3/projects/bceb73400ebb4062be126679d2f16f6d` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:52.193306 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-992c252d-8805-4f24-bad9-5341c07bae19 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:21:52.193388 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8435c20-f1e3-4930-b61b-65e7bc0f8cb6 None admin] RBAC: Authorizing `identity:get_project(project_id=bceb73400ebb4062be126679d2f16f6d)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:21:52.193388 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8435c20-f1e3-4930-b61b-65e7bc0f8cb6 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:21:52.193388 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 45/87] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Sep 25 04:21:52 2022] GET /identity/v3/projects/bceb73400ebb4062be126679d2f16f6d => generated 298 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:21:52.193388 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8435c20-f1e3-4930-b61b-65e7bc0f8cb6 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:21:52.225695 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:21:52.278957 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:21:52.305859 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-992c252d-8805-4f24-bad9-5341c07bae19 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:21:52.322218 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8435c20-f1e3-4930-b61b-65e7bc0f8cb6 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:21:52.347054 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f0bedac-7c80-461a-997d-16b9f611437d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:21:52.348132 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f0bedac-7c80-461a-997d-16b9f611437d None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:52.348427 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f0bedac-7c80-461a-997d-16b9f611437d None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:52.348736 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f0bedac-7c80-461a-997d-16b9f611437d None admin] PATH_INFO: `/v3/users/bc82e617c36e4fdfa13e159395d54e1c` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:52.350518 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f0bedac-7c80-461a-997d-16b9f611437d None admin] RBAC: Authorizing `identity:get_user(user_id=bc82e617c36e4fdfa13e159395d54e1c)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:21:52.351476 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f0bedac-7c80-461a-997d-16b9f611437d None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:21:52.357632 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 43/88] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Sep 25 04:21:52 2022] GET /identity/v3/users/bc82e617c36e4fdfa13e159395d54e1c => generated 275 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:21:52.357632 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f0bedac-7c80-461a-997d-16b9f611437d None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:21:52.365979 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8408c86c-ea2d-4cd4-a9a1-60a3b3a2fbf0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:21:52.366630 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8408c86c-ea2d-4cd4-a9a1-60a3b3a2fbf0 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:52.366914 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8408c86c-ea2d-4cd4-a9a1-60a3b3a2fbf0 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:52.367196 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8408c86c-ea2d-4cd4-a9a1-60a3b3a2fbf0 None admin] PATH_INFO: `/v3/projects/bceb73400ebb4062be126679d2f16f6d` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:52.368538 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8408c86c-ea2d-4cd4-a9a1-60a3b3a2fbf0 None admin] RBAC: Authorizing `identity:get_project(project_id=bceb73400ebb4062be126679d2f16f6d)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:21:52.369983 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8408c86c-ea2d-4cd4-a9a1-60a3b3a2fbf0 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:21:52.371633 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 46/89] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Sep 25 04:21:52 2022] GET /identity/v3/projects/bceb73400ebb4062be126679d2f16f6d => generated 298 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:21:52.372398 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8408c86c-ea2d-4cd4-a9a1-60a3b3a2fbf0 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:21:52.490723 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f0bedac-7c80-461a-997d-16b9f611437d None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:21:52.492307 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8408c86c-ea2d-4cd4-a9a1-60a3b3a2fbf0 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:21:52.492776 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:21:52.532209 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c9ecc16-ddc1-4d36-9b8b-9f31d7e57a39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:21:52.533167 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c9ecc16-ddc1-4d36-9b8b-9f31d7e57a39 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:52.533490 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c9ecc16-ddc1-4d36-9b8b-9f31d7e57a39 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:52.534099 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c9ecc16-ddc1-4d36-9b8b-9f31d7e57a39 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:52.534846 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c9ecc16-ddc1-4d36-9b8b-9f31d7e57a39 None admin] RBAC: Adding query filter params (role.id=c51fe7c5ad804040879acfb54269455f, scope.project.id=bceb73400ebb4062be126679d2f16f6d, user.id=e61aa4e8d68f455e96668bc70e0a6679) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:21:52.535163 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c9ecc16-ddc1-4d36-9b8b-9f31d7e57a39 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:21:52.536619 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c9ecc16-ddc1-4d36-9b8b-9f31d7e57a39 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:21:52.539497 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d29c671-dfc4-4929-b635-bfffa2942e04 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:21:52.540561 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d29c671-dfc4-4929-b635-bfffa2942e04 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:52.540890 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d29c671-dfc4-4929-b635-bfffa2942e04 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:52.541170 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:21:52.541344 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d29c671-dfc4-4929-b635-bfffa2942e04 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:52.543608 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 47/90] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Sep 25 04:21:52 2022] GET /identity/v3/role_assignments?user.id=e61aa4e8d68f455e96668bc70e0a6679&scope.project.id=bceb73400ebb4062be126679d2f16f6d&role.id=c51fe7c5ad804040879acfb54269455f => generated 259 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:21:52.544294 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c9ecc16-ddc1-4d36-9b8b-9f31d7e57a39 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:21:52.633919 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d29c671-dfc4-4929-b635-bfffa2942e04 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:21:52.635519 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d29c671-dfc4-4929-b635-bfffa2942e04 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:21:52.666327 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c9ecc16-ddc1-4d36-9b8b-9f31d7e57a39 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:21:52.704309 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-614ce8bc-f04d-4bf1-8635-85e78b46bd18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:21:52.704963 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-614ce8bc-f04d-4bf1-8635-85e78b46bd18 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:52.705302 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-614ce8bc-f04d-4bf1-8635-85e78b46bd18 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:52.705581 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-614ce8bc-f04d-4bf1-8635-85e78b46bd18 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:52.760555 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 44/91] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Sep 25 04:21:52 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 407 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 25 04:21:52.766212 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d29c671-dfc4-4929-b635-bfffa2942e04 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:21:52.769638 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:21:52.778392 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:21:52.780747 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-614ce8bc-f04d-4bf1-8635-85e78b46bd18 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:21:52.782472 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-614ce8bc-f04d-4bf1-8635-85e78b46bd18 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:21:52.891366 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d29c671-dfc4-4929-b635-bfffa2942e04 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:21:52.915888 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 48/92] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Sep 25 04:21:52 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 372 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 25 04:21:52.927544 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-614ce8bc-f04d-4bf1-8635-85e78b46bd18 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:21:52.930744 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aa708ea-a912-4a6a-abb3-000522403f30 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:21:52.931370 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aa708ea-a912-4a6a-abb3-000522403f30 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:52.931657 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aa708ea-a912-4a6a-abb3-000522403f30 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:52.931937 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aa708ea-a912-4a6a-abb3-000522403f30 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:52.932705 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5aa708ea-a912-4a6a-abb3-000522403f30 None admin] RBAC: Adding query filter params (role.id=8f6cc574056c4278bc9778f14dc4b411, scope.project.id=bceb73400ebb4062be126679d2f16f6d, user.id=bc82e617c36e4fdfa13e159395d54e1c) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:21:52.933027 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5aa708ea-a912-4a6a-abb3-000522403f30 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:21:52.934491 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5aa708ea-a912-4a6a-abb3-000522403f30 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:21:52.942994 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 45/93] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Sep 25 04:21:52 2022] GET /identity/v3/role_assignments?user.id=bc82e617c36e4fdfa13e159395d54e1c&scope.project.id=bceb73400ebb4062be126679d2f16f6d&role.id=8f6cc574056c4278bc9778f14dc4b411 => generated 259 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:21:52.949771 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aa708ea-a912-4a6a-abb3-000522403f30 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:21:53.090813 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-614ce8bc-f04d-4bf1-8635-85e78b46bd18 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:21:53.103321 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:21:53.110706 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:21:53.163748 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aa708ea-a912-4a6a-abb3-000522403f30 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:21:53.167502 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e6f4378-24c8-4461-8915-56fd6db95ff5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:21:53.168156 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e6f4378-24c8-4461-8915-56fd6db95ff5 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:53.168438 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e6f4378-24c8-4461-8915-56fd6db95ff5 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:53.168740 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e6f4378-24c8-4461-8915-56fd6db95ff5 None admin] PATH_INFO: `/v3/projects/bceb73400ebb4062be126679d2f16f6d` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:53.175156 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e6f4378-24c8-4461-8915-56fd6db95ff5 None admin] RBAC: Authorizing `identity:get_project(project_id=bceb73400ebb4062be126679d2f16f6d)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:21:53.176695 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e6f4378-24c8-4461-8915-56fd6db95ff5 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:21:53.178747 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 49/94] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Sep 25 04:21:52 2022] GET /identity/v3/projects/bceb73400ebb4062be126679d2f16f6d => generated 298 bytes in 262 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:21:53.185832 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e6f4378-24c8-4461-8915-56fd6db95ff5 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:21:53.241504 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3113c00-5098-4d03-8707-33d85104d440 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:21:53.242473 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3113c00-5098-4d03-8707-33d85104d440 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:53.242982 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3113c00-5098-4d03-8707-33d85104d440 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:53.243331 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3113c00-5098-4d03-8707-33d85104d440 None admin] PATH_INFO: `/v3/projects/c063d45b92f0444ebda72e3887e504f8` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:53.245906 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3113c00-5098-4d03-8707-33d85104d440 None admin] RBAC: Authorizing `identity:get_project(project_id=c063d45b92f0444ebda72e3887e504f8)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:21:53.247511 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3113c00-5098-4d03-8707-33d85104d440 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:21:53.249389 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 46/95] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Sep 25 04:21:52 2022] GET /identity/v3/projects/c063d45b92f0444ebda72e3887e504f8 => generated 312 bytes in 306 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:21:53.256997 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3113c00-5098-4d03-8707-33d85104d440 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:21:53.314329 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e6f4378-24c8-4461-8915-56fd6db95ff5 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:21:53.382164 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4c6d250-061c-40f7-a283-cde6ef58bf76 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:21:53.383391 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4c6d250-061c-40f7-a283-cde6ef58bf76 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:53.383884 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4c6d250-061c-40f7-a283-cde6ef58bf76 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:53.384489 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4c6d250-061c-40f7-a283-cde6ef58bf76 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:53.385918 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4c6d250-061c-40f7-a283-cde6ef58bf76 None admin] RBAC: Adding query filter params (role.id=02a8d263519b4ff1b511a8577293126c, scope.project.id=bceb73400ebb4062be126679d2f16f6d, user.id=bc82e617c36e4fdfa13e159395d54e1c) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:21:53.386575 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4c6d250-061c-40f7-a283-cde6ef58bf76 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:21:53.388270 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4c6d250-061c-40f7-a283-cde6ef58bf76 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:21:53.396769 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 50/96] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Sep 25 04:21:53 2022] GET /identity/v3/role_assignments?user.id=bc82e617c36e4fdfa13e159395d54e1c&scope.project.id=bceb73400ebb4062be126679d2f16f6d&role.id=02a8d263519b4ff1b511a8577293126c => generated 259 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:21:53.473248 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3113c00-5098-4d03-8707-33d85104d440 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:21:53.538525 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b1e7e0f-f70d-4c25-b5c6-b8406d9b173a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:21:53.539166 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b1e7e0f-f70d-4c25-b5c6-b8406d9b173a None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:53.539441 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b1e7e0f-f70d-4c25-b5c6-b8406d9b173a None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:53.539709 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b1e7e0f-f70d-4c25-b5c6-b8406d9b173a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:53.540450 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b1e7e0f-f70d-4c25-b5c6-b8406d9b173a None admin] RBAC: Adding query filter params (role.id=8f6cc574056c4278bc9778f14dc4b411, scope.project.id=c063d45b92f0444ebda72e3887e504f8, user.id=bc82e617c36e4fdfa13e159395d54e1c) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:21:53.540788 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b1e7e0f-f70d-4c25-b5c6-b8406d9b173a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:21:53.542272 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b1e7e0f-f70d-4c25-b5c6-b8406d9b173a None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:21:53.544348 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:21:53.554128 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 47/97] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Sep 25 04:21:53 2022] GET /identity/v3/role_assignments?user.id=bc82e617c36e4fdfa13e159395d54e1c&scope.project.id=c063d45b92f0444ebda72e3887e504f8&role.id=8f6cc574056c4278bc9778f14dc4b411 => generated 259 bytes in 298 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:21:55.198374 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d1086b6-9969-46f6-9a37-3fe1bef196e7 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:55.198888 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d1086b6-9969-46f6-9a37-3fe1bef196e7 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:55.198995 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d1086b6-9969-46f6-9a37-3fe1bef196e7 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:55.200536 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 51/98] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:21:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:21:55.205890 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de1d700-929f-4c36-99c4-8cb54a690621 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:55.206304 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de1d700-929f-4c36-99c4-8cb54a690621 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:55.206475 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de1d700-929f-4c36-99c4-8cb54a690621 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:55.234228 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-1de1d700-929f-4c36-99c4-8cb54a690621 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:21:55.235432 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-1de1d700-929f-4c36-99c4-8cb54a690621 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:21:55.324115 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 48/99] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:21:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 04:21:55.635772 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f54bbdb-56ee-4bc1-b9a6-6d3a10392bbe None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:55.636137 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f54bbdb-56ee-4bc1-b9a6-6d3a10392bbe None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:55.636333 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f54bbdb-56ee-4bc1-b9a6-6d3a10392bbe None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:55.637962 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 52/100] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:21:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:21:55.643675 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-071d467b-d807-4c57-adc3-c207da4315b1 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:55.643966 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-071d467b-d807-4c57-adc3-c207da4315b1 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:55.644245 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-071d467b-d807-4c57-adc3-c207da4315b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:55.665619 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-071d467b-d807-4c57-adc3-c207da4315b1 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:21:55.666824 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-071d467b-d807-4c57-adc3-c207da4315b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:21:55.748095 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e441abc-ef70-4176-a54d-8b16dac527b9 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:55.748771 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e441abc-ef70-4176-a54d-8b16dac527b9 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:55.749258 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e441abc-ef70-4176-a54d-8b16dac527b9 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:55.751092 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 53/101] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:21:55 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:21:55.755425 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 49/102] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:21:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 04:21:55.759673 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f4ea64-6e0d-495a-ad92-8df39c3b2b70 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:55.760180 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f4ea64-6e0d-495a-ad92-8df39c3b2b70 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:55.760693 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f4ea64-6e0d-495a-ad92-8df39c3b2b70 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:55.765884 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-071d467b-d807-4c57-adc3-c207da4315b1 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:21:55.799434 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-20f4ea64-6e0d-495a-ad92-8df39c3b2b70 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:21:55.802546 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-20f4ea64-6e0d-495a-ad92-8df39c3b2b70 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:21:55.858832 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:21:55.881083 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:21:55.933335 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-071d467b-d807-4c57-adc3-c207da4315b1 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:21:55.966034 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 54/103] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:21:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 207 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 04:21:56.005449 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a323d88f-bd5a-4df0-ae4f-d0de5f6d2088 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:21:56.006029 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a323d88f-bd5a-4df0-ae4f-d0de5f6d2088 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:56.006318 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a323d88f-bd5a-4df0-ae4f-d0de5f6d2088 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:56.006596 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a323d88f-bd5a-4df0-ae4f-d0de5f6d2088 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:56.007770 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a323d88f-bd5a-4df0-ae4f-d0de5f6d2088 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:21:56.009261 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a323d88f-bd5a-4df0-ae4f-d0de5f6d2088 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:21:56.010846 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 50/104] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:21:55 2022] GET /identity/v3/domains/default => generated 199 bytes in 245 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:21:56.016437 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20f4ea64-6e0d-495a-ad92-8df39c3b2b70 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:21:56.140428 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20f4ea64-6e0d-495a-ad92-8df39c3b2b70 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:21:56.194889 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cb4728b-3581-4f14-ac06-e9b9f171a8a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:21:56.195745 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cb4728b-3581-4f14-ac06-e9b9f171a8a0 None admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:56.196176 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cb4728b-3581-4f14-ac06-e9b9f171a8a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:56.196596 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cb4728b-3581-4f14-ac06-e9b9f171a8a0 None admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:56.197441 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cb4728b-3581-4f14-ac06-e9b9f171a8a0 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:21:56.199132 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cb4728b-3581-4f14-ac06-e9b9f171a8a0 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:21:56.236703 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 55/105] 10.222.0.25 () {60 vars in 1141 bytes} [Sun Sep 25 04:21:56 2022] POST /identity/v3/projects => generated 302 bytes in 221 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:21:56.249000 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a323d88f-bd5a-4df0-ae4f-d0de5f6d2088 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:21:56.385381 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a323d88f-bd5a-4df0-ae4f-d0de5f6d2088 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:21:56.407872 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ea32d92-64c9-49e9-8881-bf2d4b54ef7a None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:56.409083 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ea32d92-64c9-49e9-8881-bf2d4b54ef7a None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:56.409566 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ea32d92-64c9-49e9-8881-bf2d4b54ef7a None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:56.411189 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 56/106] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:21:56 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:21:56.417139 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1bdce2f-f77d-45b6-a6c4-b518d5fb2a48 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:56.417763 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1bdce2f-f77d-45b6-a6c4-b518d5fb2a48 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:56.418260 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1bdce2f-f77d-45b6-a6c4-b518d5fb2a48 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:56.425085 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:21:56.433842 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-822d8037-1423-4042-8008-310bc520f4ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:21:56.434616 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-822d8037-1423-4042-8008-310bc520f4ae None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:56.435140 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-822d8037-1423-4042-8008-310bc520f4ae None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:56.435235 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-822d8037-1423-4042-8008-310bc520f4ae None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:56.440238 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-a1bdce2f-f77d-45b6-a6c4-b518d5fb2a48 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:21:56.441857 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-a1bdce2f-f77d-45b6-a6c4-b518d5fb2a48 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:21:56.454846 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-822d8037-1423-4042-8008-310bc520f4ae None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 04:21:56.457039 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:21:56.462376 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-822d8037-1423-4042-8008-310bc520f4ae None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 25 04:21:56.462376 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:21:56.462376 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:21:56.462376 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:21:56.462376 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:21:56.462376 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:21:56.462376 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:21:56.462376 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:21:56.462376 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:21:56.462376 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:21:56.462376 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:21:56.462376 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:21:56.462376 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 04:21:56.462376 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 04:21:56.462376 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 04:21:56.462376 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 04:21:56.463267 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 04:21:56.463267 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 04:21:56.463267 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 04:21:56.463267 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 04:21:56.463267 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:21:56.463267 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:21:56.463267 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:21:56.463267 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:21:56.463267 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:21:56.463267 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:21:56.463267 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:21:56.463267 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:21:56.463267 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:21:56.463267 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:21:56.463267 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:21:56.463267 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:21:56.464087 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:21:56.464087 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:21:56.464087 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:21:56.464087 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:21:56.464087 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 04:21:56.464087 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 04:21:56.464087 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 04:21:56.464087 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 04:21:56.464087 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 04:21:56.464087 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 04:21:56.464087 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 25 04:21:56.464087 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:21:56.464087 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 51/107] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Sep 25 04:21:56 2022] GET /identity/v3/roles/reader => generated 84 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:21:56.470189 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-822d8037-1423-4042-8008-310bc520f4ae None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:21:56.552064 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 57/108] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:21:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 04:21:56.554258 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c104d24-e0bb-437c-a5a1-1c8e6c89e247 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:56.554785 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c104d24-e0bb-437c-a5a1-1c8e6c89e247 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:56.555259 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c104d24-e0bb-437c-a5a1-1c8e6c89e247 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:56.557122 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 58/109] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:21:56 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:21:56.558145 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c104d24-e0bb-437c-a5a1-1c8e6c89e247 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:21:56.649960 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-822d8037-1423-4042-8008-310bc520f4ae None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:21:56.678981 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c104d24-e0bb-437c-a5a1-1c8e6c89e247 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:21:56.688430 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b38da0a-d528-447f-bb92-1c69a0cfa0d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:21:56.689084 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b38da0a-d528-447f-bb92-1c69a0cfa0d6 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:56.690082 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b38da0a-d528-447f-bb92-1c69a0cfa0d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:56.690186 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b38da0a-d528-447f-bb92-1c69a0cfa0d6 None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:56.690830 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b38da0a-d528-447f-bb92-1c69a0cfa0d6 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:21:56.691437 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b38da0a-d528-447f-bb92-1c69a0cfa0d6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:21:56.692575 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b38da0a-d528-447f-bb92-1c69a0cfa0d6 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:21:56.702644 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 52/110] 10.222.0.25 () {56 vars in 1101 bytes} [Sun Sep 25 04:21:56 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:21:56.708267 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f72554b-fe89-425b-bfaa-456a7c47dd1a None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:56.708685 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f72554b-fe89-425b-bfaa-456a7c47dd1a None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:56.709050 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f72554b-fe89-425b-bfaa-456a7c47dd1a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:56.723856 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77cd2cd1-91f3-444e-8357-da70155d9337 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:21:56.724920 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77cd2cd1-91f3-444e-8357-da70155d9337 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:56.725381 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77cd2cd1-91f3-444e-8357-da70155d9337 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:56.725831 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77cd2cd1-91f3-444e-8357-da70155d9337 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:56.732305 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-6f72554b-fe89-425b-bfaa-456a7c47dd1a None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:21:56.734875 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-6f72554b-fe89-425b-bfaa-456a7c47dd1a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:21:56.739765 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77cd2cd1-91f3-444e-8357-da70155d9337 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 04:21:56.745192 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-77cd2cd1-91f3-444e-8357-da70155d9337 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 04:21:56.745192 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:21:56.745192 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:21:56.745192 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:21:56.745192 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:21:56.745192 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:21:56.745192 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:21:56.745192 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:21:56.745192 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:21:56.745192 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:21:56.745192 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:21:56.745192 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:21:56.745192 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 04:21:56.745192 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 04:21:56.745192 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 04:21:56.745192 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 04:21:56.746061 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 04:21:56.746061 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 04:21:56.746061 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 04:21:56.746061 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 04:21:56.746061 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:21:56.746061 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:21:56.746061 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:21:56.746061 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:21:56.746061 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:21:56.746061 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:21:56.746061 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:21:56.746061 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:21:56.746061 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:21:56.746061 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:21:56.746061 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:21:56.746061 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:21:56.746856 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:21:56.746856 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:21:56.746856 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:21:56.746856 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:21:56.746856 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 04:21:56.746856 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 04:21:56.746856 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 04:21:56.746856 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 04:21:56.746856 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 04:21:56.746856 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 04:21:56.746856 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 04:21:56.746856 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:21:56.749126 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 59/111] 10.222.0.25 () {56 vars in 1096 bytes} [Sun Sep 25 04:21:56 2022] GET /identity/v3/roles/admin => generated 83 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:21:56.750159 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77cd2cd1-91f3-444e-8357-da70155d9337 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:21:56.755822 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:21:56.797358 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:21:56.836943 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 53/112] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:21:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 04:21:56.837827 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f72554b-fe89-425b-bfaa-456a7c47dd1a None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:21:56.929925 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77cd2cd1-91f3-444e-8357-da70155d9337 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:21:56.999278 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6dd8090-a234-4cfd-ae6b-9cc86fcfbf84 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:21:56.999932 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6dd8090-a234-4cfd-ae6b-9cc86fcfbf84 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:57.000221 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6dd8090-a234-4cfd-ae6b-9cc86fcfbf84 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:57.000498 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6dd8090-a234-4cfd-ae6b-9cc86fcfbf84 None admin] PATH_INFO: `/v3/users/f525f4e8fcf2462b9f0df8ef2197b752` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:57.015450 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f72554b-fe89-425b-bfaa-456a7c47dd1a None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:21:57.026207 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6dd8090-a234-4cfd-ae6b-9cc86fcfbf84 None admin] RBAC: Authorizing `identity:get_user(user_id=f525f4e8fcf2462b9f0df8ef2197b752)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:21:57.027683 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6dd8090-a234-4cfd-ae6b-9cc86fcfbf84 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:21:57.029746 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 60/113] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Sep 25 04:21:56 2022] GET /identity/v3/users/f525f4e8fcf2462b9f0df8ef2197b752 => generated 289 bytes in 280 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:21:57.030592 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6dd8090-a234-4cfd-ae6b-9cc86fcfbf84 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:21:57.077629 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e71583a-53fb-4102-8e02-53dbdaee0e14 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:21:57.078272 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e71583a-53fb-4102-8e02-53dbdaee0e14 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:57.078925 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e71583a-53fb-4102-8e02-53dbdaee0e14 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:57.080870 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e71583a-53fb-4102-8e02-53dbdaee0e14 None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:57.081940 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e71583a-53fb-4102-8e02-53dbdaee0e14 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:21:57.082260 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e71583a-53fb-4102-8e02-53dbdaee0e14 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:21:57.083777 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e71583a-53fb-4102-8e02-53dbdaee0e14 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:21:57.107049 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 54/114] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Sep 25 04:21:56 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 270 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:21:57.113551 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e71583a-53fb-4102-8e02-53dbdaee0e14 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:21:57.150466 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:21:57.176591 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:21:57.204287 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6dd8090-a234-4cfd-ae6b-9cc86fcfbf84 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:21:57.242586 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b43458e7-3673-4824-8a91-28b3666e29bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:21:57.243242 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b43458e7-3673-4824-8a91-28b3666e29bd None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:57.243529 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b43458e7-3673-4824-8a91-28b3666e29bd None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:57.243806 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b43458e7-3673-4824-8a91-28b3666e29bd None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:57.249190 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e71583a-53fb-4102-8e02-53dbdaee0e14 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:21:57.255481 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b43458e7-3673-4824-8a91-28b3666e29bd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 04:21:57.260323 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-b43458e7-3673-4824-8a91-28b3666e29bd None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 25 04:21:57.260323 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:21:57.260323 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:21:57.260323 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:21:57.260323 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:21:57.260323 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:21:57.260323 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:21:57.260323 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:21:57.260323 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:21:57.260323 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:21:57.260323 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:21:57.260323 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:21:57.260323 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 04:21:57.260323 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 04:21:57.260323 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 04:21:57.260323 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 04:21:57.261219 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 04:21:57.261219 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 04:21:57.261219 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 04:21:57.261219 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 04:21:57.261219 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:21:57.261219 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:21:57.261219 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:21:57.261219 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:21:57.261219 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:21:57.261219 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:21:57.261219 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:21:57.261219 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:21:57.261219 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:21:57.261219 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:21:57.261219 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:21:57.261219 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:21:57.261999 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:21:57.261999 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:21:57.261999 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:21:57.261999 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:21:57.261999 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 04:21:57.261999 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 04:21:57.261999 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 04:21:57.261999 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 04:21:57.261999 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 04:21:57.261999 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 04:21:57.261999 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 25 04:21:57.261999 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:21:57.261999 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 61/115] 10.222.0.25 () {56 vars in 1114 bytes} [Sun Sep 25 04:21:57 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 231 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:21:57.263249 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b43458e7-3673-4824-8a91-28b3666e29bd None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:21:57.292299 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32edcf99-2529-4c78-88f0-6ec7ccf16769 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:21:57.292954 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32edcf99-2529-4c78-88f0-6ec7ccf16769 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:57.293299 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32edcf99-2529-4c78-88f0-6ec7ccf16769 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:57.293503 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32edcf99-2529-4c78-88f0-6ec7ccf16769 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:57.369293 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32edcf99-2529-4c78-88f0-6ec7ccf16769 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:21:57.370775 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32edcf99-2529-4c78-88f0-6ec7ccf16769 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:21:57.371459 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b43458e7-3673-4824-8a91-28b3666e29bd None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:21:57.399357 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:21:57.413660 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1cfc39f-a17b-4b25-b2c5-8dd66c30cb50 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:21:57.414308 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1cfc39f-a17b-4b25-b2c5-8dd66c30cb50 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:57.414589 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1cfc39f-a17b-4b25-b2c5-8dd66c30cb50 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:57.414859 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1cfc39f-a17b-4b25-b2c5-8dd66c30cb50 None admin] PATH_INFO: `/v3/users/e61aa4e8d68f455e96668bc70e0a6679` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:57.415999 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1cfc39f-a17b-4b25-b2c5-8dd66c30cb50 None admin] RBAC: Authorizing `identity:get_user(user_id=e61aa4e8d68f455e96668bc70e0a6679)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:21:57.417964 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1cfc39f-a17b-4b25-b2c5-8dd66c30cb50 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:21:57.419587 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:21:57.419904 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 62/116] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Sep 25 04:21:57 2022] GET /identity/v3/users/e61aa4e8d68f455e96668bc70e0a6679 => generated 247 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:21:57.425555 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1cfc39f-a17b-4b25-b2c5-8dd66c30cb50 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:21:57.492695 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 55/117] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Sep 25 04:21:57 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 385 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 25 04:21:57.493826 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f07abfd8-e1c2-4af7-8770-956e051b2db3 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:57.494119 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f07abfd8-e1c2-4af7-8770-956e051b2db3 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:57.494396 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f07abfd8-e1c2-4af7-8770-956e051b2db3 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:57.495799 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 56/118] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:21:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:21:57.500251 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f07abfd8-e1c2-4af7-8770-956e051b2db3 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:21:57.543311 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1cfc39f-a17b-4b25-b2c5-8dd66c30cb50 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:21:57.582453 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c2ed548-0e85-4b49-a30b-a57797d38b54 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:21:57.583572 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c2ed548-0e85-4b49-a30b-a57797d38b54 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:57.584088 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c2ed548-0e85-4b49-a30b-a57797d38b54 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:57.584577 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c2ed548-0e85-4b49-a30b-a57797d38b54 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:57.596418 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c2ed548-0e85-4b49-a30b-a57797d38b54 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 04:21:57.601662 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-9c2ed548-0e85-4b49-a30b-a57797d38b54 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 25 04:21:57.601662 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:21:57.601662 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:21:57.601662 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:21:57.601662 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:21:57.601662 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:21:57.601662 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:21:57.601662 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:21:57.601662 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:21:57.601662 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:21:57.601662 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:21:57.601662 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:21:57.601662 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 04:21:57.601662 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 04:21:57.601662 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 04:21:57.601662 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 04:21:57.602563 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 04:21:57.602563 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 04:21:57.602563 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 04:21:57.602563 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 04:21:57.602563 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:21:57.602563 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:21:57.602563 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:21:57.602563 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:21:57.602563 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:21:57.602563 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:21:57.602563 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:21:57.602563 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:21:57.602563 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:21:57.602563 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:21:57.602563 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:21:57.602563 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:21:57.605591 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:21:57.605591 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:21:57.605591 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:21:57.605591 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:21:57.605591 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 04:21:57.605591 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 04:21:57.605591 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 04:21:57.605591 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 04:21:57.605591 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 04:21:57.605591 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 04:21:57.605591 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 25 04:21:57.605591 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:21:57.605591 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 63/119] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Sep 25 04:21:57 2022] GET /identity/v3/roles/member => generated 84 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:21:57.605591 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c2ed548-0e85-4b49-a30b-a57797d38b54 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:21:57.620085 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f07abfd8-e1c2-4af7-8770-956e051b2db3 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:21:57.656264 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:21:57.660374 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0673e7a-8596-48d0-88eb-e27661a1aa5f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:21:57.661391 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0673e7a-8596-48d0-88eb-e27661a1aa5f None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:57.661826 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0673e7a-8596-48d0-88eb-e27661a1aa5f None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:57.662249 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0673e7a-8596-48d0-88eb-e27661a1aa5f None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:57.663514 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0673e7a-8596-48d0-88eb-e27661a1aa5f None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:21:57.664117 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0673e7a-8596-48d0-88eb-e27661a1aa5f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:21:57.665805 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0673e7a-8596-48d0-88eb-e27661a1aa5f None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:21:57.675053 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 57/120] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Sep 25 04:21:57 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:21:57.675869 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0673e7a-8596-48d0-88eb-e27661a1aa5f None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:21:57.678571 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:21:57.714756 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c2ed548-0e85-4b49-a30b-a57797d38b54 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:21:57.752968 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ac764f0-31f1-41f7-aba2-ca8230856acd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:21:57.753541 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ac764f0-31f1-41f7-aba2-ca8230856acd None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:57.753832 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ac764f0-31f1-41f7-aba2-ca8230856acd None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:57.754114 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ac764f0-31f1-41f7-aba2-ca8230856acd None admin] PATH_INFO: `/v3/projects/bceb73400ebb4062be126679d2f16f6d` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:57.755264 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ac764f0-31f1-41f7-aba2-ca8230856acd None admin] RBAC: Authorizing `identity:get_project(project_id=bceb73400ebb4062be126679d2f16f6d)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:21:57.756760 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ac764f0-31f1-41f7-aba2-ca8230856acd None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:21:57.758704 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 64/121] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Sep 25 04:21:57 2022] GET /identity/v3/projects/bceb73400ebb4062be126679d2f16f6d => generated 298 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:21:57.762816 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-308f54bb-8be1-45e4-ae8d-1180d7c20793 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:57.762816 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-308f54bb-8be1-45e4-ae8d-1180d7c20793 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:57.762816 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-308f54bb-8be1-45e4-ae8d-1180d7c20793 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:57.778571 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-308f54bb-8be1-45e4-ae8d-1180d7c20793 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:21:57.779775 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-308f54bb-8be1-45e4-ae8d-1180d7c20793 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:21:57.789614 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0673e7a-8596-48d0-88eb-e27661a1aa5f None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:21:57.825453 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91218d55-718c-4c2c-824f-c7c9686c575f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:21:57.826087 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91218d55-718c-4c2c-824f-c7c9686c575f None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:57.826368 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91218d55-718c-4c2c-824f-c7c9686c575f None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:57.826643 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91218d55-718c-4c2c-824f-c7c9686c575f None admin] PATH_INFO: `/v3/projects/bceb73400ebb4062be126679d2f16f6d` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:57.827882 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91218d55-718c-4c2c-824f-c7c9686c575f None admin] RBAC: Authorizing `identity:get_project(project_id=bceb73400ebb4062be126679d2f16f6d)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:21:57.829463 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91218d55-718c-4c2c-824f-c7c9686c575f None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:21:57.831767 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 58/122] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Sep 25 04:21:57 2022] GET /identity/v3/projects/bceb73400ebb4062be126679d2f16f6d => generated 298 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:21:57.832653 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91218d55-718c-4c2c-824f-c7c9686c575f None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:21:57.853630 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 65/123] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:21:57 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 04:21:57.865561 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-308f54bb-8be1-45e4-ae8d-1180d7c20793 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:21:57.916739 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:21:57.921581 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:21:57.956344 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91218d55-718c-4c2c-824f-c7c9686c575f None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:21:57.987519 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-308f54bb-8be1-45e4-ae8d-1180d7c20793 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:21:57.994903 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-870b4e25-4d86-4781-bd26-d92f66c4131e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:21:57.995527 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-870b4e25-4d86-4781-bd26-d92f66c4131e None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:57.995810 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-870b4e25-4d86-4781-bd26-d92f66c4131e None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:57.996083 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-870b4e25-4d86-4781-bd26-d92f66c4131e None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:57.996856 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-870b4e25-4d86-4781-bd26-d92f66c4131e None admin] RBAC: Adding query filter params (name=member) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:21:57.997166 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-870b4e25-4d86-4781-bd26-d92f66c4131e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:21:57.999017 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-870b4e25-4d86-4781-bd26-d92f66c4131e None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:21:58.013905 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 59/124] 10.222.0.25 () {56 vars in 1101 bytes} [Sun Sep 25 04:21:57 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:21:58.013905 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-870b4e25-4d86-4781-bd26-d92f66c4131e None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:21:58.039198 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d84d5d19-9e57-4e51-abcd-7a5dbaa71adc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:21:58.040589 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d84d5d19-9e57-4e51-abcd-7a5dbaa71adc None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:58.041332 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d84d5d19-9e57-4e51-abcd-7a5dbaa71adc None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:58.041808 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d84d5d19-9e57-4e51-abcd-7a5dbaa71adc None admin] PATH_INFO: `/v3/users/bc82e617c36e4fdfa13e159395d54e1c` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:58.043790 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d84d5d19-9e57-4e51-abcd-7a5dbaa71adc None admin] RBAC: Authorizing `identity:get_user(user_id=bc82e617c36e4fdfa13e159395d54e1c)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:21:58.046518 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d84d5d19-9e57-4e51-abcd-7a5dbaa71adc None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:21:58.049208 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 66/125] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Sep 25 04:21:57 2022] GET /identity/v3/users/bc82e617c36e4fdfa13e159395d54e1c => generated 275 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:21:58.051103 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d84d5d19-9e57-4e51-abcd-7a5dbaa71adc None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:21:58.129327 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-870b4e25-4d86-4781-bd26-d92f66c4131e None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:21:58.160688 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:21:58.165869 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d84d5d19-9e57-4e51-abcd-7a5dbaa71adc None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:21:58.166633 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7a95d24-cc5a-4177-9827-6df48e5d7188 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:21:58.167314 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7a95d24-cc5a-4177-9827-6df48e5d7188 None admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:58.167599 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7a95d24-cc5a-4177-9827-6df48e5d7188 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:58.167909 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7a95d24-cc5a-4177-9827-6df48e5d7188 None admin] PATH_INFO: `/v3/projects/bceb73400ebb4062be126679d2f16f6d/users/e61aa4e8d68f455e96668bc70e0a6679/roles/c51fe7c5ad804040879acfb54269455f` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:58.169864 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7a95d24-cc5a-4177-9827-6df48e5d7188 None admin] RBAC: Authorizing `identity:create_grant(project_id=bceb73400ebb4062be126679d2f16f6d, user_id=e61aa4e8d68f455e96668bc70e0a6679, role_id=c51fe7c5ad804040879acfb54269455f)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:21:58.171321 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7a95d24-cc5a-4177-9827-6df48e5d7188 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:21:58.184977 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 60/126] 10.222.0.25 () {58 vars in 1439 bytes} [Sun Sep 25 04:21:58 2022] PUT /identity/v3/projects/bceb73400ebb4062be126679d2f16f6d/users/e61aa4e8d68f455e96668bc70e0a6679/roles/c51fe7c5ad804040879acfb54269455f => generated 0 bytes in 175 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:21:58.185778 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7a95d24-cc5a-4177-9827-6df48e5d7188 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:21:58.193799 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:21:58.217190 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6eb4884-ae79-4e57-92fb-582a31b11e1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:21:58.217858 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6eb4884-ae79-4e57-92fb-582a31b11e1b None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:58.218268 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6eb4884-ae79-4e57-92fb-582a31b11e1b None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:58.218544 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6eb4884-ae79-4e57-92fb-582a31b11e1b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:58.219279 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6eb4884-ae79-4e57-92fb-582a31b11e1b None admin] RBAC: Adding query filter params (role.id=0e4fa3b3402d463da90ddfcd17be7c17, scope.project.id=bceb73400ebb4062be126679d2f16f6d, user.id=f525f4e8fcf2462b9f0df8ef2197b752) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:21:58.219591 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6eb4884-ae79-4e57-92fb-582a31b11e1b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:21:58.222311 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6eb4884-ae79-4e57-92fb-582a31b11e1b None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:21:58.229083 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 67/127] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Sep 25 04:21:58 2022] GET /identity/v3/role_assignments?user.id=f525f4e8fcf2462b9f0df8ef2197b752&scope.project.id=bceb73400ebb4062be126679d2f16f6d&role.id=0e4fa3b3402d463da90ddfcd17be7c17 => generated 259 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:21:58.230063 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6eb4884-ae79-4e57-92fb-582a31b11e1b None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:21:58.331715 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7a95d24-cc5a-4177-9827-6df48e5d7188 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:21:58.372621 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6eb4884-ae79-4e57-92fb-582a31b11e1b None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:21:58.418537 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2369ef0-19d0-4f26-bdea-18131a51f18f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:21:58.419527 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2369ef0-19d0-4f26-bdea-18131a51f18f None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:58.420002 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2369ef0-19d0-4f26-bdea-18131a51f18f None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:58.420488 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2369ef0-19d0-4f26-bdea-18131a51f18f None admin] PATH_INFO: `/v3/projects/bceb73400ebb4062be126679d2f16f6d` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:58.422071 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2369ef0-19d0-4f26-bdea-18131a51f18f None admin] RBAC: Authorizing `identity:get_project(project_id=bceb73400ebb4062be126679d2f16f6d)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:21:58.422934 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae54c76a-3e75-438a-b46c-eb92e34416e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:21:58.424154 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae54c76a-3e75-438a-b46c-eb92e34416e2 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:58.424440 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae54c76a-3e75-438a-b46c-eb92e34416e2 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:58.424753 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae54c76a-3e75-438a-b46c-eb92e34416e2 None admin] PATH_INFO: `/v3/users/bc82e617c36e4fdfa13e159395d54e1c` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:58.425392 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2369ef0-19d0-4f26-bdea-18131a51f18f None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:21:58.427390 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 68/128] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Sep 25 04:21:58 2022] GET /identity/v3/projects/bceb73400ebb4062be126679d2f16f6d => generated 298 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:21:58.428602 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2369ef0-19d0-4f26-bdea-18131a51f18f None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:21:58.433948 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae54c76a-3e75-438a-b46c-eb92e34416e2 None admin] RBAC: Authorizing `identity:get_user(user_id=bc82e617c36e4fdfa13e159395d54e1c)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:21:58.433948 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae54c76a-3e75-438a-b46c-eb92e34416e2 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:21:58.433948 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 61/129] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Sep 25 04:21:58 2022] GET /identity/v3/users/bc82e617c36e4fdfa13e159395d54e1c => generated 275 bytes in 247 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:21:58.437698 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae54c76a-3e75-438a-b46c-eb92e34416e2 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:21:58.514398 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:21:58.536850 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:21:58.577995 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2369ef0-19d0-4f26-bdea-18131a51f18f None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:21:58.590163 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae54c76a-3e75-438a-b46c-eb92e34416e2 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:21:58.631674 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cbdcea5-8ea9-499f-9786-ff9716e2ba06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:21:58.632769 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cbdcea5-8ea9-499f-9786-ff9716e2ba06 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:58.633835 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cbdcea5-8ea9-499f-9786-ff9716e2ba06 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:58.634437 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cbdcea5-8ea9-499f-9786-ff9716e2ba06 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:58.650688 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f555ea2-7fea-4d1b-9254-c145cb105583 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:21:58.652004 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f555ea2-7fea-4d1b-9254-c145cb105583 None admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:58.652602 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f555ea2-7fea-4d1b-9254-c145cb105583 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:58.654125 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f555ea2-7fea-4d1b-9254-c145cb105583 None admin] PATH_INFO: `/v3/projects/bceb73400ebb4062be126679d2f16f6d/users/bc82e617c36e4fdfa13e159395d54e1c/roles/02a8d263519b4ff1b511a8577293126c` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:58.659156 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f555ea2-7fea-4d1b-9254-c145cb105583 None admin] RBAC: Authorizing `identity:create_grant(project_id=bceb73400ebb4062be126679d2f16f6d, user_id=bc82e617c36e4fdfa13e159395d54e1c, role_id=02a8d263519b4ff1b511a8577293126c)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:21:58.659917 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cbdcea5-8ea9-499f-9786-ff9716e2ba06 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 04:21:58.664520 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f555ea2-7fea-4d1b-9254-c145cb105583 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:21:58.669977 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-2cbdcea5-8ea9-499f-9786-ff9716e2ba06 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 25 04:21:58.669977 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:21:58.669977 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:21:58.669977 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:21:58.669977 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:21:58.669977 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:21:58.669977 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:21:58.669977 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:21:58.669977 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:21:58.669977 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:21:58.669977 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:21:58.669977 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:21:58.669977 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 04:21:58.669977 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 04:21:58.669977 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 04:21:58.669977 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 04:21:58.670864 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 04:21:58.670864 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 04:21:58.670864 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 04:21:58.670864 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 04:21:58.670864 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:21:58.670864 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:21:58.670864 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:21:58.670864 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:21:58.670864 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:21:58.670864 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:21:58.670864 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:21:58.670864 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:21:58.670864 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:21:58.670864 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:21:58.670864 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:21:58.670864 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:21:58.671652 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:21:58.671652 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:21:58.671652 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:21:58.671652 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:21:58.671652 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 04:21:58.671652 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 04:21:58.671652 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 04:21:58.671652 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 04:21:58.671652 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 04:21:58.671652 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 04:21:58.671652 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 25 04:21:58.671652 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:21:58.676538 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 69/130] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Sep 25 04:21:58 2022] GET /identity/v3/roles/member => generated 84 bytes in 248 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:21:58.677781 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cbdcea5-8ea9-499f-9786-ff9716e2ba06 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:21:58.690381 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 62/131] 10.222.0.25 () {58 vars in 1439 bytes} [Sun Sep 25 04:21:58 2022] PUT /identity/v3/projects/bceb73400ebb4062be126679d2f16f6d/users/bc82e617c36e4fdfa13e159395d54e1c/roles/02a8d263519b4ff1b511a8577293126c => generated 0 bytes in 253 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:21:58.691442 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f555ea2-7fea-4d1b-9254-c145cb105583 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:21:58.894061 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cbdcea5-8ea9-499f-9786-ff9716e2ba06 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:21:58.905820 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f555ea2-7fea-4d1b-9254-c145cb105583 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:21:58.913025 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:21:58.944383 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:21:58.967114 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cd6a62d-70e3-4009-9a26-1125c9838a34 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:21:58.969727 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cd6a62d-70e3-4009-9a26-1125c9838a34 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:58.970019 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cd6a62d-70e3-4009-9a26-1125c9838a34 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:58.970301 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cd6a62d-70e3-4009-9a26-1125c9838a34 None admin] PATH_INFO: `/v3/projects/bceb73400ebb4062be126679d2f16f6d` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:58.973424 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-667b4e95-999c-4d55-9d7e-08eb1e7e31e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:21:58.974050 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-667b4e95-999c-4d55-9d7e-08eb1e7e31e0 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:58.974330 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-667b4e95-999c-4d55-9d7e-08eb1e7e31e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:58.974600 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-667b4e95-999c-4d55-9d7e-08eb1e7e31e0 None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:58.975283 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-667b4e95-999c-4d55-9d7e-08eb1e7e31e0 None admin] RBAC: Adding query filter params (name=member) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:21:58.975590 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-667b4e95-999c-4d55-9d7e-08eb1e7e31e0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:21:58.977012 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-667b4e95-999c-4d55-9d7e-08eb1e7e31e0 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:21:58.979153 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cd6a62d-70e3-4009-9a26-1125c9838a34 None admin] RBAC: Authorizing `identity:get_project(project_id=bceb73400ebb4062be126679d2f16f6d)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:21:58.980671 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cd6a62d-70e3-4009-9a26-1125c9838a34 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:21:58.982785 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 70/132] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Sep 25 04:21:58 2022] GET /identity/v3/projects/bceb73400ebb4062be126679d2f16f6d => generated 298 bytes in 305 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:21:58.989101 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cd6a62d-70e3-4009-9a26-1125c9838a34 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:21:58.993263 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 63/133] 10.222.0.25 () {56 vars in 1101 bytes} [Sun Sep 25 04:21:58 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 302 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:21:59.000258 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-667b4e95-999c-4d55-9d7e-08eb1e7e31e0 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:21:59.136966 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cd6a62d-70e3-4009-9a26-1125c9838a34 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:21:59.139986 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-667b4e95-999c-4d55-9d7e-08eb1e7e31e0 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:21:59.177307 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ddcff4a-2ee2-49bf-8c77-7f9b881189bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:21:59.177999 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ddcff4a-2ee2-49bf-8c77-7f9b881189bf None admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:59.178354 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ddcff4a-2ee2-49bf-8c77-7f9b881189bf None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:59.178703 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ddcff4a-2ee2-49bf-8c77-7f9b881189bf None admin] PATH_INFO: `/v3/projects/bceb73400ebb4062be126679d2f16f6d/users/bc82e617c36e4fdfa13e159395d54e1c/roles/8f6cc574056c4278bc9778f14dc4b411` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:59.180973 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ddcff4a-2ee2-49bf-8c77-7f9b881189bf None admin] RBAC: Authorizing `identity:create_grant(project_id=bceb73400ebb4062be126679d2f16f6d, user_id=bc82e617c36e4fdfa13e159395d54e1c, role_id=8f6cc574056c4278bc9778f14dc4b411)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:21:59.182531 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ddcff4a-2ee2-49bf-8c77-7f9b881189bf None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:21:59.184507 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff46c1c6-67e4-4ffa-bcfe-ae74257ad6f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:21:59.185249 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff46c1c6-67e4-4ffa-bcfe-ae74257ad6f3 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:59.185599 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff46c1c6-67e4-4ffa-bcfe-ae74257ad6f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:59.185943 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff46c1c6-67e4-4ffa-bcfe-ae74257ad6f3 None admin] PATH_INFO: `/v3/users/bc82e617c36e4fdfa13e159395d54e1c` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:59.187151 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff46c1c6-67e4-4ffa-bcfe-ae74257ad6f3 None admin] RBAC: Authorizing `identity:get_user(user_id=bc82e617c36e4fdfa13e159395d54e1c)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:21:59.188681 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff46c1c6-67e4-4ffa-bcfe-ae74257ad6f3 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:21:59.191455 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 64/134] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Sep 25 04:21:58 2022] GET /identity/v3/users/bc82e617c36e4fdfa13e159395d54e1c => generated 275 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:21:59.196788 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7273cb36-7a33-491c-80c5-243cdd269122 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:59.197227 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7273cb36-7a33-491c-80c5-243cdd269122 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:59.197616 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7273cb36-7a33-491c-80c5-243cdd269122 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:59.199315 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 65/135] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:21:59 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:21:59.207040 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7bc30ff-27ff-4ca0-ad3e-76ba74c60abc None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:59.207443 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7bc30ff-27ff-4ca0-ad3e-76ba74c60abc None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:59.207792 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7bc30ff-27ff-4ca0-ad3e-76ba74c60abc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:59.214186 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 71/136] 10.222.0.25 () {58 vars in 1439 bytes} [Sun Sep 25 04:21:58 2022] PUT /identity/v3/projects/bceb73400ebb4062be126679d2f16f6d/users/bc82e617c36e4fdfa13e159395d54e1c/roles/8f6cc574056c4278bc9778f14dc4b411 => generated 0 bytes in 226 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:21:59.215127 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ddcff4a-2ee2-49bf-8c77-7f9b881189bf None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:21:59.235954 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-e7bc30ff-27ff-4ca0-ad3e-76ba74c60abc None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:21:59.242321 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-e7bc30ff-27ff-4ca0-ad3e-76ba74c60abc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:21:59.312734 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:21:59.333165 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:21:59.357210 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 66/137] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:21:59 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 152 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 25 04:21:59.436581 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ddcff4a-2ee2-49bf-8c77-7f9b881189bf None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:21:59.499038 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d083971-411f-4f4c-b69c-519fb211b354 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:21:59.499723 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d083971-411f-4f4c-b69c-519fb211b354 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:59.500055 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d083971-411f-4f4c-b69c-519fb211b354 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:59.500286 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d083971-411f-4f4c-b69c-519fb211b354 None admin] PATH_INFO: `/v3/projects/c063d45b92f0444ebda72e3887e504f8` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:59.502263 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d083971-411f-4f4c-b69c-519fb211b354 None admin] RBAC: Authorizing `identity:get_project(project_id=c063d45b92f0444ebda72e3887e504f8)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:21:59.503789 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d083971-411f-4f4c-b69c-519fb211b354 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:21:59.505615 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 72/138] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Sep 25 04:21:59 2022] GET /identity/v3/projects/c063d45b92f0444ebda72e3887e504f8 => generated 312 bytes in 291 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:21:59.511536 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7bc30ff-27ff-4ca0-ad3e-76ba74c60abc None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:21:59.634960 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7bc30ff-27ff-4ca0-ad3e-76ba74c60abc None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:21:59.671950 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d779f941-504a-47f5-9739-cf8fa900aa44 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:21:59.672629 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d779f941-504a-47f5-9739-cf8fa900aa44 None admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:59.672939 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d779f941-504a-47f5-9739-cf8fa900aa44 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:59.673215 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d779f941-504a-47f5-9739-cf8fa900aa44 None admin] PATH_INFO: `/v3/projects/c063d45b92f0444ebda72e3887e504f8/users/bc82e617c36e4fdfa13e159395d54e1c/roles/8f6cc574056c4278bc9778f14dc4b411` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:59.675512 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d779f941-504a-47f5-9739-cf8fa900aa44 None admin] RBAC: Authorizing `identity:create_grant(project_id=c063d45b92f0444ebda72e3887e504f8, user_id=bc82e617c36e4fdfa13e159395d54e1c, role_id=8f6cc574056c4278bc9778f14dc4b411)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:21:59.676988 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d779f941-504a-47f5-9739-cf8fa900aa44 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:21:59.694127 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 67/139] 10.222.0.25 () {58 vars in 1439 bytes} [Sun Sep 25 04:21:59 2022] PUT /identity/v3/projects/c063d45b92f0444ebda72e3887e504f8/users/bc82e617c36e4fdfa13e159395d54e1c/roles/8f6cc574056c4278bc9778f14dc4b411 => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:21:59.823285 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d083971-411f-4f4c-b69c-519fb211b354 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:21:59.942328 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d083971-411f-4f4c-b69c-519fb211b354 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:21:59.944955 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:21:59.986496 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-731f9a22-5cfa-4918-ac99-c99dbe3c67e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:21:59.987414 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-731f9a22-5cfa-4918-ac99-c99dbe3c67e8 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:21:59.987682 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-731f9a22-5cfa-4918-ac99-c99dbe3c67e8 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:21:59.987979 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-731f9a22-5cfa-4918-ac99-c99dbe3c67e8 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:21:59.989152 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-731f9a22-5cfa-4918-ac99-c99dbe3c67e8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:21:59.990605 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-731f9a22-5cfa-4918-ac99-c99dbe3c67e8 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:21:59.992178 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 73/140] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:21:59 2022] GET /identity/v3/domains/default => generated 199 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:21:59.997609 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d779f941-504a-47f5-9739-cf8fa900aa44 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:00.049667 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:00.106319 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d779f941-504a-47f5-9739-cf8fa900aa44 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:00.141806 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-971aaec1-0f02-480c-822e-60ae86f3b834 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:00.142969 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-971aaec1-0f02-480c-822e-60ae86f3b834 None admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:00.143245 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-971aaec1-0f02-480c-822e-60ae86f3b834 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:00.143516 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-971aaec1-0f02-480c-822e-60ae86f3b834 None admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:00.144193 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-971aaec1-0f02-480c-822e-60ae86f3b834 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:00.145689 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-971aaec1-0f02-480c-822e-60ae86f3b834 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:00.177423 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 68/141] 10.222.0.25 () {60 vars in 1132 bytes} [Sun Sep 25 04:21:59 2022] POST /identity/v3/users => generated 283 bytes in 180 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:22:00.996614 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-592c43e9-12e0-4e57-8acb-5ea750741cdf None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:00.997258 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-592c43e9-12e0-4e57-8acb-5ea750741cdf None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:00.997258 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-592c43e9-12e0-4e57-8acb-5ea750741cdf None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:00.998725 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 74/142] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:22:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:22:01.005561 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf2ef49-9931-4f66-8952-419e50d0a5a8 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:01.006455 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf2ef49-9931-4f66-8952-419e50d0a5a8 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:01.006941 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf2ef49-9931-4f66-8952-419e50d0a5a8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:01.027935 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-ecf2ef49-9931-4f66-8952-419e50d0a5a8 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:22:01.029527 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-ecf2ef49-9931-4f66-8952-419e50d0a5a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:22:01.110116 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 69/143] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:22:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 25 04:22:01.444812 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-592c43e9-12e0-4e57-8acb-5ea750741cdf None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:01.577343 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-592c43e9-12e0-4e57-8acb-5ea750741cdf None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:01.616195 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9f8c4b0-08a7-4c5a-9da5-9f6668d06a40 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:01.616882 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f8c4b0-08a7-4c5a-9da5-9f6668d06a40 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:01.617166 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f8c4b0-08a7-4c5a-9da5-9f6668d06a40 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:01.617473 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f8c4b0-08a7-4c5a-9da5-9f6668d06a40 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:01.631964 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9f8c4b0-08a7-4c5a-9da5-9f6668d06a40 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 04:22:01.641584 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-f9f8c4b0-08a7-4c5a-9da5-9f6668d06a40 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 25 04:22:01.641584 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:22:01.641584 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:22:01.641584 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:22:01.641584 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:22:01.641584 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:22:01.641584 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:22:01.641584 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:22:01.641584 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:22:01.641584 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:22:01.641584 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:22:01.641584 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:22:01.641584 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 04:22:01.641584 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 04:22:01.641584 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 04:22:01.641584 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 04:22:01.643331 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 04:22:01.643331 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 04:22:01.643331 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 04:22:01.643331 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 04:22:01.643331 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:22:01.643331 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:22:01.643331 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:22:01.643331 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:22:01.643331 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:22:01.643331 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:22:01.643331 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:22:01.643331 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:22:01.643331 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:22:01.643331 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:22:01.643331 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:22:01.643331 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:22:01.644149 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:22:01.644149 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:22:01.644149 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:22:01.644149 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:22:01.644149 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 04:22:01.644149 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 04:22:01.644149 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 04:22:01.644149 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 04:22:01.644149 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 04:22:01.644149 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 04:22:01.644149 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 25 04:22:01.644149 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:22:01.644149 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 75/144] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Sep 25 04:22:01 2022] GET /identity/v3/roles/reader => generated 84 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:22:01.652690 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecf2ef49-9931-4f66-8952-419e50d0a5a8 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:01.704062 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:01.804307 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecf2ef49-9931-4f66-8952-419e50d0a5a8 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:01.844314 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8063fcd1-9bad-42dd-b4cf-61ed04e11e06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:01.844986 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8063fcd1-9bad-42dd-b4cf-61ed04e11e06 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:01.845301 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8063fcd1-9bad-42dd-b4cf-61ed04e11e06 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:01.846089 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8063fcd1-9bad-42dd-b4cf-61ed04e11e06 None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:01.846771 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8063fcd1-9bad-42dd-b4cf-61ed04e11e06 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:01.847079 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8063fcd1-9bad-42dd-b4cf-61ed04e11e06 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:01.848470 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8063fcd1-9bad-42dd-b4cf-61ed04e11e06 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:01.860189 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 70/145] 10.222.0.25 () {56 vars in 1101 bytes} [Sun Sep 25 04:22:01 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:01.866192 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9f8c4b0-08a7-4c5a-9da5-9f6668d06a40 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:01.868406 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49c9f85b-c89e-4b65-bd3f-4e0c8561a6ca None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:01.868731 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49c9f85b-c89e-4b65-bd3f-4e0c8561a6ca None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:01.868998 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49c9f85b-c89e-4b65-bd3f-4e0c8561a6ca None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:01.870570 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 71/146] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:22:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:22:01.877855 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7357556b-71c4-410d-83ed-aae3f15e4e4d None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:01.878147 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7357556b-71c4-410d-83ed-aae3f15e4e4d None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:01.878430 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7357556b-71c4-410d-83ed-aae3f15e4e4d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:01.900756 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-7357556b-71c4-410d-83ed-aae3f15e4e4d None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:22:01.901973 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-7357556b-71c4-410d-83ed-aae3f15e4e4d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:22:01.931153 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:02.014929 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 72/147] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:22:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 25 04:22:02.039119 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50239705-0632-4b4b-a0ae-68e47e9898e4 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:02.039422 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50239705-0632-4b4b-a0ae-68e47e9898e4 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:02.039695 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50239705-0632-4b4b-a0ae-68e47e9898e4 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:02.041067 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 73/148] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:22:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 25 04:22:02.046076 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-309e3b39-ed4f-4fa4-9765-7da97941783c None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:02.046363 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-309e3b39-ed4f-4fa4-9765-7da97941783c None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:02.046635 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-309e3b39-ed4f-4fa4-9765-7da97941783c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:02.068698 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9f8c4b0-08a7-4c5a-9da5-9f6668d06a40 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:02.081147 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-309e3b39-ed4f-4fa4-9765-7da97941783c None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:22:02.082432 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-309e3b39-ed4f-4fa4-9765-7da97941783c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:22:02.116993 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:02.136174 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-079bf60e-55fe-4071-80c6-3804c3027b02 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:02.136879 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-079bf60e-55fe-4071-80c6-3804c3027b02 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:02.137164 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-079bf60e-55fe-4071-80c6-3804c3027b02 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:02.137504 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-079bf60e-55fe-4071-80c6-3804c3027b02 None admin] PATH_INFO: `/v3/users/f525f4e8fcf2462b9f0df8ef2197b752` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:02.141331 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-079bf60e-55fe-4071-80c6-3804c3027b02 None admin] RBAC: Authorizing `identity:get_user(user_id=f525f4e8fcf2462b9f0df8ef2197b752)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:02.143370 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-079bf60e-55fe-4071-80c6-3804c3027b02 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:02.145014 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 76/149] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Sep 25 04:22:01 2022] GET /identity/v3/users/f525f4e8fcf2462b9f0df8ef2197b752 => generated 289 bytes in 279 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:02.150625 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-079bf60e-55fe-4071-80c6-3804c3027b02 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:02.221547 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 74/150] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:22:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 25 04:22:02.317370 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:02.323701 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-079bf60e-55fe-4071-80c6-3804c3027b02 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:02.373461 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af577932-9a15-4fd0-aaa7-7f290f94f9b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:02.374112 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af577932-9a15-4fd0-aaa7-7f290f94f9b4 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:02.374394 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af577932-9a15-4fd0-aaa7-7f290f94f9b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:02.374667 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af577932-9a15-4fd0-aaa7-7f290f94f9b4 None admin] PATH_INFO: `/v3/projects/bceb73400ebb4062be126679d2f16f6d` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:02.375846 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af577932-9a15-4fd0-aaa7-7f290f94f9b4 None admin] RBAC: Authorizing `identity:get_project(project_id=bceb73400ebb4062be126679d2f16f6d)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:02.377596 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af577932-9a15-4fd0-aaa7-7f290f94f9b4 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:02.379234 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 77/151] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Sep 25 04:22:02 2022] GET /identity/v3/projects/bceb73400ebb4062be126679d2f16f6d => generated 298 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:22:02.384992 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-309e3b39-ed4f-4fa4-9765-7da97941783c None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:02.503623 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af577932-9a15-4fd0-aaa7-7f290f94f9b4 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:02.518079 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-309e3b39-ed4f-4fa4-9765-7da97941783c None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:02.579799 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89649aa2-6d52-4a4a-94d9-4bfdfd8412f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:02.580665 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89649aa2-6d52-4a4a-94d9-4bfdfd8412f9 None admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:02.581060 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89649aa2-6d52-4a4a-94d9-4bfdfd8412f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:02.581346 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89649aa2-6d52-4a4a-94d9-4bfdfd8412f9 None admin] PATH_INFO: `/v3/projects/bceb73400ebb4062be126679d2f16f6d/users/f525f4e8fcf2462b9f0df8ef2197b752/roles/0e4fa3b3402d463da90ddfcd17be7c17` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:02.583373 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89649aa2-6d52-4a4a-94d9-4bfdfd8412f9 None admin] RBAC: Authorizing `identity:create_grant(project_id=bceb73400ebb4062be126679d2f16f6d, user_id=f525f4e8fcf2462b9f0df8ef2197b752, role_id=0e4fa3b3402d463da90ddfcd17be7c17)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:02.588663 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89649aa2-6d52-4a4a-94d9-4bfdfd8412f9 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:02.615805 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 75/152] 10.222.0.25 () {58 vars in 1439 bytes} [Sun Sep 25 04:22:02 2022] PUT /identity/v3/projects/bceb73400ebb4062be126679d2f16f6d/users/f525f4e8fcf2462b9f0df8ef2197b752/roles/0e4fa3b3402d463da90ddfcd17be7c17 => generated 0 bytes in 231 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:22:02.656084 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af577932-9a15-4fd0-aaa7-7f290f94f9b4 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:02.703504 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-857bac20-7b76-4c96-9327-3fe080404fa6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:02.704506 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-857bac20-7b76-4c96-9327-3fe080404fa6 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:02.705134 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-857bac20-7b76-4c96-9327-3fe080404fa6 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:02.705641 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-857bac20-7b76-4c96-9327-3fe080404fa6 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:02.718497 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-857bac20-7b76-4c96-9327-3fe080404fa6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 04:22:02.725383 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-857bac20-7b76-4c96-9327-3fe080404fa6 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 25 04:22:02.725383 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:22:02.725383 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:22:02.725383 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:22:02.725383 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:22:02.725383 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:22:02.725383 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:22:02.725383 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:22:02.725383 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:22:02.725383 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:22:02.725383 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:22:02.725383 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:22:02.725383 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 04:22:02.725383 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 04:22:02.725383 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 04:22:02.725383 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 04:22:02.726396 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 04:22:02.726396 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 04:22:02.726396 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 04:22:02.726396 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 04:22:02.726396 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:22:02.726396 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:22:02.726396 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:22:02.726396 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:22:02.726396 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:22:02.726396 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:22:02.726396 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:22:02.726396 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:22:02.726396 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:22:02.726396 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:22:02.726396 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:22:02.726396 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:22:02.727194 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:22:02.727194 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:22:02.727194 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:22:02.727194 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:22:02.727194 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 04:22:02.727194 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 04:22:02.727194 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 04:22:02.727194 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 04:22:02.727194 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 04:22:02.727194 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 04:22:02.727194 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 25 04:22:02.727194 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:22:02.732927 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 78/153] 10.222.0.25 () {56 vars in 1114 bytes} [Sun Sep 25 04:22:02 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 230 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 04:22:02.738932 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89649aa2-6d52-4a4a-94d9-4bfdfd8412f9 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:02.752199 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:02.871906 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89649aa2-6d52-4a4a-94d9-4bfdfd8412f9 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:02.911765 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-857bac20-7b76-4c96-9327-3fe080404fa6 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:02.918608 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a95ff62e-3ba7-427b-b78b-076434259af3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:02.918608 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a95ff62e-3ba7-427b-b78b-076434259af3 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:02.918608 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a95ff62e-3ba7-427b-b78b-076434259af3 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:02.918608 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a95ff62e-3ba7-427b-b78b-076434259af3 None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:02.918984 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a95ff62e-3ba7-427b-b78b-076434259af3 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:02.918984 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a95ff62e-3ba7-427b-b78b-076434259af3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:02.919652 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a95ff62e-3ba7-427b-b78b-076434259af3 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:02.930304 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 76/154] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Sep 25 04:22:02 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:02.940423 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a95ff62e-3ba7-427b-b78b-076434259af3 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:02.979057 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:03.017989 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:03.054795 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a95ff62e-3ba7-427b-b78b-076434259af3 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:03.098596 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82674444-2182-42ae-808d-a53e1db3b373 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:03.102425 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82674444-2182-42ae-808d-a53e1db3b373 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:03.102705 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82674444-2182-42ae-808d-a53e1db3b373 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:03.102979 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82674444-2182-42ae-808d-a53e1db3b373 None admin] PATH_INFO: `/v3/users/bc82e617c36e4fdfa13e159395d54e1c` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:03.104913 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82674444-2182-42ae-808d-a53e1db3b373 None admin] RBAC: Authorizing `identity:get_user(user_id=bc82e617c36e4fdfa13e159395d54e1c)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:03.106355 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82674444-2182-42ae-808d-a53e1db3b373 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:03.107954 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 77/155] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Sep 25 04:22:02 2022] GET /identity/v3/users/bc82e617c36e4fdfa13e159395d54e1c => generated 275 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:22:03.114914 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82674444-2182-42ae-808d-a53e1db3b373 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:03.120759 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-857bac20-7b76-4c96-9327-3fe080404fa6 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:03.198150 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-490bedad-c10b-4cc0-987d-4158b9224aaa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:03.198798 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-490bedad-c10b-4cc0-987d-4158b9224aaa None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:03.199076 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-490bedad-c10b-4cc0-987d-4158b9224aaa None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:03.199347 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-490bedad-c10b-4cc0-987d-4158b9224aaa None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:03.221988 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-490bedad-c10b-4cc0-987d-4158b9224aaa None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 04:22:03.227727 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-490bedad-c10b-4cc0-987d-4158b9224aaa None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 04:22:03.227727 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:22:03.227727 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:22:03.227727 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:22:03.227727 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:22:03.227727 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:22:03.227727 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:22:03.227727 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:22:03.227727 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:22:03.227727 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:22:03.227727 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:22:03.227727 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:22:03.227727 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 04:22:03.227727 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 04:22:03.227727 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 04:22:03.227727 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 04:22:03.228784 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 04:22:03.228784 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 04:22:03.228784 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 04:22:03.228784 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 04:22:03.228784 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:22:03.228784 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:22:03.228784 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:22:03.228784 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:22:03.228784 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:22:03.228784 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:22:03.228784 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:22:03.228784 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:22:03.228784 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:22:03.228784 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:22:03.228784 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:22:03.228784 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:22:03.230031 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:22:03.230031 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:22:03.230031 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:22:03.230031 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:22:03.230031 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 04:22:03.230031 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 04:22:03.230031 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 04:22:03.230031 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 04:22:03.230031 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 04:22:03.230031 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 04:22:03.230031 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 04:22:03.230031 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:22:03.230031 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 79/156] 10.222.0.25 () {56 vars in 1096 bytes} [Sun Sep 25 04:22:02 2022] GET /identity/v3/roles/admin => generated 83 bytes in 318 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 04:22:03.237196 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-490bedad-c10b-4cc0-987d-4158b9224aaa None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:03.289636 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82674444-2182-42ae-808d-a53e1db3b373 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:03.325522 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6cfd9c9-97ad-4ce2-aa3b-f16788fa734c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:03.326146 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6cfd9c9-97ad-4ce2-aa3b-f16788fa734c None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:03.326425 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6cfd9c9-97ad-4ce2-aa3b-f16788fa734c None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:03.326694 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6cfd9c9-97ad-4ce2-aa3b-f16788fa734c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:03.341986 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:03.406836 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:03.425681 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6cfd9c9-97ad-4ce2-aa3b-f16788fa734c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:03.427160 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6cfd9c9-97ad-4ce2-aa3b-f16788fa734c None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:03.434810 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-490bedad-c10b-4cc0-987d-4158b9224aaa None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:03.513433 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e80e094-3300-4540-a6a1-e1af0325ef8b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:03.514072 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e80e094-3300-4540-a6a1-e1af0325ef8b None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:03.514353 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e80e094-3300-4540-a6a1-e1af0325ef8b None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:03.514628 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e80e094-3300-4540-a6a1-e1af0325ef8b None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:03.515428 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e80e094-3300-4540-a6a1-e1af0325ef8b None admin] RBAC: Adding query filter params (name=admin) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:03.515740 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e80e094-3300-4540-a6a1-e1af0325ef8b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:03.517261 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e80e094-3300-4540-a6a1-e1af0325ef8b None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:03.533430 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 80/157] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Sep 25 04:22:03 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 297 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:22:03.539396 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e80e094-3300-4540-a6a1-e1af0325ef8b None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:03.543488 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 78/158] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Sep 25 04:22:03 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 429 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 25 04:22:03.549058 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6cfd9c9-97ad-4ce2-aa3b-f16788fa734c None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:03.603628 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:03.709339 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6cfd9c9-97ad-4ce2-aa3b-f16788fa734c None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:03.718491 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e80e094-3300-4540-a6a1-e1af0325ef8b None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:03.766020 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76871868-3de0-46b8-afc2-acc65b918fa5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:03.766640 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76871868-3de0-46b8-afc2-acc65b918fa5 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:03.766917 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76871868-3de0-46b8-afc2-acc65b918fa5 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:03.767763 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76871868-3de0-46b8-afc2-acc65b918fa5 None admin] PATH_INFO: `/v3/projects/bceb73400ebb4062be126679d2f16f6d` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:03.769330 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76871868-3de0-46b8-afc2-acc65b918fa5 None admin] RBAC: Authorizing `identity:get_project(project_id=bceb73400ebb4062be126679d2f16f6d)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:03.770927 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76871868-3de0-46b8-afc2-acc65b918fa5 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:03.772612 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 79/159] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Sep 25 04:22:03 2022] GET /identity/v3/projects/bceb73400ebb4062be126679d2f16f6d => generated 298 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:22:03.779805 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76871868-3de0-46b8-afc2-acc65b918fa5 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:03.800182 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-649461a7-d23e-40d3-8d2f-a731ce62f84a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:03.800902 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-649461a7-d23e-40d3-8d2f-a731ce62f84a None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:03.801186 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-649461a7-d23e-40d3-8d2f-a731ce62f84a None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:03.801462 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-649461a7-d23e-40d3-8d2f-a731ce62f84a None admin] PATH_INFO: `/v3/users/e61aa4e8d68f455e96668bc70e0a6679` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:03.803278 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-649461a7-d23e-40d3-8d2f-a731ce62f84a None admin] RBAC: Authorizing `identity:get_user(user_id=e61aa4e8d68f455e96668bc70e0a6679)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:03.804867 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-649461a7-d23e-40d3-8d2f-a731ce62f84a None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:03.806869 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 81/160] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Sep 25 04:22:03 2022] GET /identity/v3/users/e61aa4e8d68f455e96668bc70e0a6679 => generated 247 bytes in 268 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:22:03.812215 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-649461a7-d23e-40d3-8d2f-a731ce62f84a None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:03.852953 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:03.926961 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76871868-3de0-46b8-afc2-acc65b918fa5 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:03.935471 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:03.965045 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b10658ce-10a0-4cd6-83cd-8799df0ff003 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:03.965656 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b10658ce-10a0-4cd6-83cd-8799df0ff003 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:03.965947 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b10658ce-10a0-4cd6-83cd-8799df0ff003 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:03.966199 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b10658ce-10a0-4cd6-83cd-8799df0ff003 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:03.966929 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b10658ce-10a0-4cd6-83cd-8799df0ff003 None admin] RBAC: Adding query filter params (role.id=02a8d263519b4ff1b511a8577293126c, scope.project.id=bceb73400ebb4062be126679d2f16f6d, user.id=bc82e617c36e4fdfa13e159395d54e1c) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:03.967331 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b10658ce-10a0-4cd6-83cd-8799df0ff003 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:03.968794 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b10658ce-10a0-4cd6-83cd-8799df0ff003 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:03.975620 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 80/161] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Sep 25 04:22:03 2022] GET /identity/v3/role_assignments?user.id=bc82e617c36e4fdfa13e159395d54e1c&scope.project.id=bceb73400ebb4062be126679d2f16f6d&role.id=02a8d263519b4ff1b511a8577293126c => generated 608 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:22:03.976814 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faac72f7-8e53-46b4-a752-63dfd1143810 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:03.977209 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faac72f7-8e53-46b4-a752-63dfd1143810 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:03.977488 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faac72f7-8e53-46b4-a752-63dfd1143810 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:03.978931 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 81/162] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:22:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:22:03.984235 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de5b1501-ef7d-40f3-b18b-77c7908b3b20 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:03.984518 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de5b1501-ef7d-40f3-b18b-77c7908b3b20 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:03.984839 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de5b1501-ef7d-40f3-b18b-77c7908b3b20 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:04.011579 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-de5b1501-ef7d-40f3-b18b-77c7908b3b20 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:22:04.012833 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-de5b1501-ef7d-40f3-b18b-77c7908b3b20 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:22:04.017040 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-649461a7-d23e-40d3-8d2f-a731ce62f84a None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:04.097757 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69f6846c-eb1f-4f2c-b22a-2072f09a0d89 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:04.098389 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69f6846c-eb1f-4f2c-b22a-2072f09a0d89 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:04.098677 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69f6846c-eb1f-4f2c-b22a-2072f09a0d89 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:04.098953 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69f6846c-eb1f-4f2c-b22a-2072f09a0d89 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:04.145564 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 82/163] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:22:03 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 25 04:22:04.146975 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9c4ccab-0ff8-4969-bb31-5403d9fcee02 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:04.147263 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9c4ccab-0ff8-4969-bb31-5403d9fcee02 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:04.147531 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9c4ccab-0ff8-4969-bb31-5403d9fcee02 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:04.148868 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 83/164] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:22:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:22:04.155159 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aee1f2ed-265f-4136-a4a1-b591264c0ebe None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:04.155440 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aee1f2ed-265f-4136-a4a1-b591264c0ebe None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:04.155709 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aee1f2ed-265f-4136-a4a1-b591264c0ebe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:04.190543 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-aee1f2ed-265f-4136-a4a1-b591264c0ebe None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:22:04.191886 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-aee1f2ed-265f-4136-a4a1-b591264c0ebe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:22:04.211507 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69f6846c-eb1f-4f2c-b22a-2072f09a0d89 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:04.213080 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69f6846c-eb1f-4f2c-b22a-2072f09a0d89 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:04.254058 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:04.344102 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 84/165] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:22:04 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 04:22:04.396035 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 82/166] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Sep 25 04:22:03 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 584 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 25 04:22:04.401834 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aee1f2ed-265f-4136-a4a1-b591264c0ebe None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:04.445036 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:04.535675 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aee1f2ed-265f-4136-a4a1-b591264c0ebe None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:04.581860 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b01360b-20d7-4bdb-970e-5bf6a79e81c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:04.582512 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b01360b-20d7-4bdb-970e-5bf6a79e81c9 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:04.582790 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b01360b-20d7-4bdb-970e-5bf6a79e81c9 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:04.583112 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b01360b-20d7-4bdb-970e-5bf6a79e81c9 None admin] PATH_INFO: `/v3/projects/bceb73400ebb4062be126679d2f16f6d` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:04.584258 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b01360b-20d7-4bdb-970e-5bf6a79e81c9 None admin] RBAC: Authorizing `identity:get_project(project_id=bceb73400ebb4062be126679d2f16f6d)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:04.585731 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b01360b-20d7-4bdb-970e-5bf6a79e81c9 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:04.587578 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 85/167] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Sep 25 04:22:04 2022] GET /identity/v3/projects/bceb73400ebb4062be126679d2f16f6d => generated 298 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:04.593195 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69f6846c-eb1f-4f2c-b22a-2072f09a0d89 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:04.743689 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69f6846c-eb1f-4f2c-b22a-2072f09a0d89 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:04.769778 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:04.792087 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ed487b7-418c-496d-ac3c-9f48db96b665 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:04.792759 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ed487b7-418c-496d-ac3c-9f48db96b665 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:04.793053 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ed487b7-418c-496d-ac3c-9f48db96b665 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:04.793318 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ed487b7-418c-496d-ac3c-9f48db96b665 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:04.794062 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ed487b7-418c-496d-ac3c-9f48db96b665 None admin] RBAC: Adding query filter params (role.id=c51fe7c5ad804040879acfb54269455f, scope.project.id=bceb73400ebb4062be126679d2f16f6d, user.id=e61aa4e8d68f455e96668bc70e0a6679) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:04.794376 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ed487b7-418c-496d-ac3c-9f48db96b665 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:04.795860 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ed487b7-418c-496d-ac3c-9f48db96b665 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:04.804001 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 83/168] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Sep 25 04:22:04 2022] GET /identity/v3/role_assignments?user.id=e61aa4e8d68f455e96668bc70e0a6679&scope.project.id=bceb73400ebb4062be126679d2f16f6d&role.id=c51fe7c5ad804040879acfb54269455f => generated 608 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:04.866050 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b01360b-20d7-4bdb-970e-5bf6a79e81c9 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:04.913413 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ed487b7-418c-496d-ac3c-9f48db96b665 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:05.025186 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:05.031475 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b01360b-20d7-4bdb-970e-5bf6a79e81c9 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:05.089331 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ed487b7-418c-496d-ac3c-9f48db96b665 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:05.098402 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c108d6d-0b46-43f0-87cf-c87962e51515 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:05.099069 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c108d6d-0b46-43f0-87cf-c87962e51515 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:05.099349 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c108d6d-0b46-43f0-87cf-c87962e51515 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:05.099622 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c108d6d-0b46-43f0-87cf-c87962e51515 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:05.125489 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c108d6d-0b46-43f0-87cf-c87962e51515 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 04:22:05.131831 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-2c108d6d-0b46-43f0-87cf-c87962e51515 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 25 04:22:05.131831 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:22:05.131831 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:22:05.131831 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:22:05.131831 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:22:05.131831 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:22:05.131831 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:22:05.131831 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:22:05.131831 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:22:05.131831 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:22:05.131831 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:22:05.131831 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:22:05.131831 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 04:22:05.131831 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 04:22:05.131831 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 04:22:05.131831 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 04:22:05.133163 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 04:22:05.133163 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 04:22:05.133163 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 04:22:05.133163 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 04:22:05.133163 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:22:05.133163 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:22:05.133163 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:22:05.133163 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:22:05.133163 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:22:05.133163 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:22:05.133163 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:22:05.133163 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:22:05.133163 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:22:05.133163 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:22:05.133163 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:22:05.133163 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:22:05.134220 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:22:05.134220 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:22:05.134220 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:22:05.134220 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:22:05.134220 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 04:22:05.134220 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 04:22:05.134220 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 04:22:05.134220 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 04:22:05.134220 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 04:22:05.134220 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 04:22:05.134220 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 25 04:22:05.134220 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:22:05.134220 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 86/169] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Sep 25 04:22:04 2022] GET /identity/v3/roles/member => generated 84 bytes in 268 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:22:05.147205 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c108d6d-0b46-43f0-87cf-c87962e51515 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:05.166103 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e78ecef8-176b-447a-ae7c-9f1b386b39be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:05.167290 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e78ecef8-176b-447a-ae7c-9f1b386b39be None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:05.167793 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e78ecef8-176b-447a-ae7c-9f1b386b39be None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:05.168416 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e78ecef8-176b-447a-ae7c-9f1b386b39be None admin] PATH_INFO: `/v3/roles/member` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:05.187668 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e78ecef8-176b-447a-ae7c-9f1b386b39be None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 04:22:05.197920 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-e78ecef8-176b-447a-ae7c-9f1b386b39be None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 25 04:22:05.197920 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:22:05.197920 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:22:05.197920 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:22:05.197920 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:22:05.197920 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:22:05.197920 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:22:05.197920 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:22:05.197920 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:22:05.197920 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:22:05.197920 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:22:05.197920 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:22:05.197920 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 04:22:05.197920 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 04:22:05.197920 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 04:22:05.197920 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 04:22:05.198879 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 04:22:05.198879 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 04:22:05.198879 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 04:22:05.198879 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 04:22:05.198879 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:22:05.198879 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:22:05.198879 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:22:05.198879 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:22:05.198879 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:22:05.198879 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:22:05.198879 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:22:05.198879 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:22:05.198879 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:22:05.198879 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:22:05.198879 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:22:05.198879 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:22:05.199682 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:22:05.199682 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:22:05.199682 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:22:05.199682 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:22:05.199682 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 04:22:05.199682 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 04:22:05.199682 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 04:22:05.199682 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 04:22:05.199682 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 04:22:05.199682 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 04:22:05.199682 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 25 04:22:05.199682 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:22:05.203498 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 84/170] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Sep 25 04:22:04 2022] GET /identity/v3/roles/member => generated 84 bytes in 290 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:22:05.208962 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e78ecef8-176b-447a-ae7c-9f1b386b39be None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:05.290768 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:05.337172 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c108d6d-0b46-43f0-87cf-c87962e51515 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:05.392000 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e78ecef8-176b-447a-ae7c-9f1b386b39be None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:05.403158 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92c04358-6678-498d-9de4-771d2f1c023c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:05.403908 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c04358-6678-498d-9de4-771d2f1c023c None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:05.404189 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c04358-6678-498d-9de4-771d2f1c023c None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:05.404464 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c04358-6678-498d-9de4-771d2f1c023c None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:05.405169 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92c04358-6678-498d-9de4-771d2f1c023c None admin] RBAC: Adding query filter params (name=member) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:05.405474 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92c04358-6678-498d-9de4-771d2f1c023c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:05.407222 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92c04358-6678-498d-9de4-771d2f1c023c None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:05.425446 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 87/171] 10.222.0.25 () {56 vars in 1101 bytes} [Sun Sep 25 04:22:05 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 277 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:22:05.425446 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-243ad250-c828-43be-a42a-ce15c5a89ea9 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:05.425724 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-243ad250-c828-43be-a42a-ce15c5a89ea9 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:05.425890 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-243ad250-c828-43be-a42a-ce15c5a89ea9 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:05.427372 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 88/172] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:22:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:22:05.432410 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-243ad250-c828-43be-a42a-ce15c5a89ea9 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:05.441010 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4c8c00b-b38d-4970-aabf-405bfb461c2b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:05.441865 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c8c00b-b38d-4970-aabf-405bfb461c2b None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:05.442294 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c8c00b-b38d-4970-aabf-405bfb461c2b None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:05.442713 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c8c00b-b38d-4970-aabf-405bfb461c2b None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:05.443552 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4c8c00b-b38d-4970-aabf-405bfb461c2b None admin] RBAC: Adding query filter params (name=member) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:05.444005 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4c8c00b-b38d-4970-aabf-405bfb461c2b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:05.445575 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4c8c00b-b38d-4970-aabf-405bfb461c2b None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:05.456160 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 85/173] 10.222.0.25 () {56 vars in 1101 bytes} [Sun Sep 25 04:22:05 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 248 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:22:05.461797 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4ede2cb-7241-4336-8e89-f4889e96e5f9 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:05.462308 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4ede2cb-7241-4336-8e89-f4889e96e5f9 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:05.462741 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4ede2cb-7241-4336-8e89-f4889e96e5f9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:05.480181 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:05.491376 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-a4ede2cb-7241-4336-8e89-f4889e96e5f9 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:22:05.495834 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-a4ede2cb-7241-4336-8e89-f4889e96e5f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:22:05.620274 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-243ad250-c828-43be-a42a-ce15c5a89ea9 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:05.628535 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 86/174] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:22:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 04:22:05.629446 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4ede2cb-7241-4336-8e89-f4889e96e5f9 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:05.683421 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-864e6877-69cc-43b3-91ce-211fb27cd948 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:05.684487 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-864e6877-69cc-43b3-91ce-211fb27cd948 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:05.684794 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-864e6877-69cc-43b3-91ce-211fb27cd948 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:05.685070 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-864e6877-69cc-43b3-91ce-211fb27cd948 None admin] PATH_INFO: `/v3/users/bc82e617c36e4fdfa13e159395d54e1c` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:05.686784 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-864e6877-69cc-43b3-91ce-211fb27cd948 None admin] RBAC: Authorizing `identity:get_user(user_id=bc82e617c36e4fdfa13e159395d54e1c)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:05.690895 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-864e6877-69cc-43b3-91ce-211fb27cd948 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:05.692518 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 89/175] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Sep 25 04:22:05 2022] GET /identity/v3/users/bc82e617c36e4fdfa13e159395d54e1c => generated 275 bytes in 260 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:05.693031 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:05.699896 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-864e6877-69cc-43b3-91ce-211fb27cd948 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:05.822131 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4ede2cb-7241-4336-8e89-f4889e96e5f9 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:05.869743 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:05.883452 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-864e6877-69cc-43b3-91ce-211fb27cd948 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:05.888550 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16601e95-fefb-4aab-a812-49be48c8142d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:05.889233 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16601e95-fefb-4aab-a812-49be48c8142d None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:05.889517 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16601e95-fefb-4aab-a812-49be48c8142d None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:05.889797 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16601e95-fefb-4aab-a812-49be48c8142d None admin] PATH_INFO: `/v3/users/bc82e617c36e4fdfa13e159395d54e1c` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:05.895046 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16601e95-fefb-4aab-a812-49be48c8142d None admin] RBAC: Authorizing `identity:get_user(user_id=bc82e617c36e4fdfa13e159395d54e1c)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:05.896585 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16601e95-fefb-4aab-a812-49be48c8142d None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:05.898488 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 87/176] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Sep 25 04:22:05 2022] GET /identity/v3/users/bc82e617c36e4fdfa13e159395d54e1c => generated 275 bytes in 269 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:05.899631 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad436c0-d687-4246-ba04-c867a657ae59 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:05.899915 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad436c0-d687-4246-ba04-c867a657ae59 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:05.900190 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad436c0-d687-4246-ba04-c867a657ae59 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:05.906116 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 88/177] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:22:05 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:22:05.906807 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cad436c0-d687-4246-ba04-c867a657ae59 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:05.936113 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f524807-2628-43e7-b99b-1c3285fe6642 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:05.937283 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f524807-2628-43e7-b99b-1c3285fe6642 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:05.937687 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f524807-2628-43e7-b99b-1c3285fe6642 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:05.938059 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f524807-2628-43e7-b99b-1c3285fe6642 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:06.015332 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f524807-2628-43e7-b99b-1c3285fe6642 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:06.018105 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f524807-2628-43e7-b99b-1c3285fe6642 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:06.060656 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cad436c0-d687-4246-ba04-c867a657ae59 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:06.062768 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:06.129615 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:06.138169 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df2234e5-dc2d-4b28-8089-419ad1215093 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:06.139062 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df2234e5-dc2d-4b28-8089-419ad1215093 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:06.139561 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df2234e5-dc2d-4b28-8089-419ad1215093 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:06.140739 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df2234e5-dc2d-4b28-8089-419ad1215093 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:06.156843 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 90/178] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Sep 25 04:22:05 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 457 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 25 04:22:06.158986 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e83b9a6-04a7-41cc-9471-47ef41975007 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:06.161825 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e83b9a6-04a7-41cc-9471-47ef41975007 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:06.162335 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e83b9a6-04a7-41cc-9471-47ef41975007 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:06.182657 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-1e83b9a6-04a7-41cc-9471-47ef41975007 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:22:06.184208 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-1e83b9a6-04a7-41cc-9471-47ef41975007 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:22:06.230747 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df2234e5-dc2d-4b28-8089-419ad1215093 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:06.232546 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df2234e5-dc2d-4b28-8089-419ad1215093 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:06.268664 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 91/179] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:22:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 04:22:06.268664 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e83b9a6-04a7-41cc-9471-47ef41975007 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:06.360302 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e83b9a6-04a7-41cc-9471-47ef41975007 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:06.371163 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 89/180] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Sep 25 04:22:05 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 464 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 25 04:22:06.372744 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df2234e5-dc2d-4b28-8089-419ad1215093 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:06.375573 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:06.397015 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:06.398291 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-796c4557-f2e7-4532-a8bc-0f54dbf13067 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:06.399158 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-796c4557-f2e7-4532-a8bc-0f54dbf13067 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:06.399577 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-796c4557-f2e7-4532-a8bc-0f54dbf13067 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:06.400050 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-796c4557-f2e7-4532-a8bc-0f54dbf13067 None admin] PATH_INFO: `/v3/projects/c063d45b92f0444ebda72e3887e504f8` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:06.401534 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-796c4557-f2e7-4532-a8bc-0f54dbf13067 None admin] RBAC: Authorizing `identity:get_project(project_id=c063d45b92f0444ebda72e3887e504f8)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:06.403464 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-796c4557-f2e7-4532-a8bc-0f54dbf13067 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:06.405229 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 92/181] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Sep 25 04:22:06 2022] GET /identity/v3/projects/c063d45b92f0444ebda72e3887e504f8 => generated 312 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:06.406449 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74664306-1806-47bc-9a19-f296688d5f02 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:06.407047 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74664306-1806-47bc-9a19-f296688d5f02 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:06.407163 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74664306-1806-47bc-9a19-f296688d5f02 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:06.409575 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 93/182] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:22:06 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:22:06.414489 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74664306-1806-47bc-9a19-f296688d5f02 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:06.502119 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df2234e5-dc2d-4b28-8089-419ad1215093 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:06.534545 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74664306-1806-47bc-9a19-f296688d5f02 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:06.547787 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1631d4ea-d5a5-4eb0-848f-2d9984d535ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:06.548519 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1631d4ea-d5a5-4eb0-848f-2d9984d535ee None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:06.549045 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1631d4ea-d5a5-4eb0-848f-2d9984d535ee None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:06.549322 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1631d4ea-d5a5-4eb0-848f-2d9984d535ee None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:06.560833 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1631d4ea-d5a5-4eb0-848f-2d9984d535ee None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 04:22:06.565759 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-1631d4ea-d5a5-4eb0-848f-2d9984d535ee None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 04:22:06.565759 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:22:06.565759 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:22:06.565759 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:22:06.565759 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:22:06.565759 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:22:06.565759 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:22:06.565759 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:22:06.565759 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:22:06.565759 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:22:06.565759 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:22:06.565759 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:22:06.565759 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 04:22:06.565759 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 04:22:06.565759 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 04:22:06.565759 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 04:22:06.566844 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 04:22:06.566844 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 04:22:06.566844 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 04:22:06.566844 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 04:22:06.566844 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:22:06.566844 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:22:06.566844 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:22:06.566844 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:22:06.566844 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:22:06.566844 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:22:06.566844 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:22:06.566844 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:22:06.566844 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:22:06.566844 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:22:06.566844 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:22:06.566844 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:22:06.567834 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:22:06.567834 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:22:06.567834 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:22:06.567834 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:22:06.567834 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 04:22:06.567834 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 04:22:06.567834 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 04:22:06.567834 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 04:22:06.567834 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 04:22:06.567834 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 04:22:06.567834 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 04:22:06.567834 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:22:06.567834 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 90/183] 10.222.0.25 () {56 vars in 1096 bytes} [Sun Sep 25 04:22:06 2022] GET /identity/v3/roles/admin => generated 83 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:22:06.567834 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1631d4ea-d5a5-4eb0-848f-2d9984d535ee None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:06.585066 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe3c2d36-d9cf-475c-b6ef-2e2db3b391b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:06.586194 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe3c2d36-d9cf-475c-b6ef-2e2db3b391b6 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:06.586632 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe3c2d36-d9cf-475c-b6ef-2e2db3b391b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:06.587324 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe3c2d36-d9cf-475c-b6ef-2e2db3b391b6 None admin] PATH_INFO: `/v3/projects/bceb73400ebb4062be126679d2f16f6d` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:06.589011 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe3c2d36-d9cf-475c-b6ef-2e2db3b391b6 None admin] RBAC: Authorizing `identity:get_project(project_id=bceb73400ebb4062be126679d2f16f6d)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:06.591144 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe3c2d36-d9cf-475c-b6ef-2e2db3b391b6 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:06.593461 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 94/184] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Sep 25 04:22:06 2022] GET /identity/v3/projects/bceb73400ebb4062be126679d2f16f6d => generated 298 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:06.595996 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e10b7eb3-7170-412f-b6fc-02d75a5fa77b None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:06.596726 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e10b7eb3-7170-412f-b6fc-02d75a5fa77b None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:06.597388 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e10b7eb3-7170-412f-b6fc-02d75a5fa77b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:06.619147 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-e10b7eb3-7170-412f-b6fc-02d75a5fa77b None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:22:06.620351 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-e10b7eb3-7170-412f-b6fc-02d75a5fa77b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:22:06.633925 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:06.671916 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:06.676992 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1631d4ea-d5a5-4eb0-848f-2d9984d535ee None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:06.689726 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 95/185] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:22:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 04:22:06.690708 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e10b7eb3-7170-412f-b6fc-02d75a5fa77b None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:06.717793 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6024afd3-2e45-4430-9e12-d56a2ff4d116 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:06.718406 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6024afd3-2e45-4430-9e12-d56a2ff4d116 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:06.718688 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6024afd3-2e45-4430-9e12-d56a2ff4d116 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:06.718958 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6024afd3-2e45-4430-9e12-d56a2ff4d116 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:06.719680 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6024afd3-2e45-4430-9e12-d56a2ff4d116 None admin] RBAC: Adding query filter params (role.id=8f6cc574056c4278bc9778f14dc4b411, scope.project.id=c063d45b92f0444ebda72e3887e504f8, user.id=bc82e617c36e4fdfa13e159395d54e1c) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:06.719992 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6024afd3-2e45-4430-9e12-d56a2ff4d116 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:06.722217 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6024afd3-2e45-4430-9e12-d56a2ff4d116 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:06.728436 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 91/186] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Sep 25 04:22:06 2022] GET /identity/v3/role_assignments?user.id=bc82e617c36e4fdfa13e159395d54e1c&scope.project.id=c063d45b92f0444ebda72e3887e504f8&role.id=8f6cc574056c4278bc9778f14dc4b411 => generated 608 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:06.728738 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6024afd3-2e45-4430-9e12-d56a2ff4d116 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:06.808780 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e10b7eb3-7170-412f-b6fc-02d75a5fa77b None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:06.851699 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82665ac3-9132-4833-a96c-600ae9a93a76 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:06.853033 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82665ac3-9132-4833-a96c-600ae9a93a76 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:06.853945 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82665ac3-9132-4833-a96c-600ae9a93a76 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:06.854400 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82665ac3-9132-4833-a96c-600ae9a93a76 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:06.870601 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82665ac3-9132-4833-a96c-600ae9a93a76 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 04:22:06.870797 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6024afd3-2e45-4430-9e12-d56a2ff4d116 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:06.877578 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-82665ac3-9132-4833-a96c-600ae9a93a76 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 25 04:22:06.877578 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:22:06.877578 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:22:06.877578 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:22:06.877578 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:22:06.877578 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:22:06.877578 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:22:06.877578 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:22:06.877578 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:22:06.877578 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:22:06.877578 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:22:06.877578 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:22:06.877578 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 04:22:06.877578 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 04:22:06.877578 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 04:22:06.877578 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 04:22:06.878560 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 04:22:06.878560 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 04:22:06.878560 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 04:22:06.878560 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 04:22:06.878560 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:22:06.878560 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:22:06.878560 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:22:06.878560 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:22:06.878560 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:22:06.878560 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:22:06.878560 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:22:06.878560 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:22:06.878560 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:22:06.878560 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:22:06.878560 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:22:06.878560 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:22:06.879368 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:22:06.879368 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:22:06.879368 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:22:06.879368 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:22:06.879368 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 04:22:06.879368 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 04:22:06.879368 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 04:22:06.879368 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 04:22:06.879368 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 04:22:06.879368 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 04:22:06.879368 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 25 04:22:06.879368 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:22:06.887491 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 96/187] 10.222.0.25 () {56 vars in 1114 bytes} [Sun Sep 25 04:22:06 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:22:06.887491 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82665ac3-9132-4833-a96c-600ae9a93a76 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:06.919907 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:06.922841 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22318d62-0b69-4656-87f7-9a167fb10905 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:06.924830 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22318d62-0b69-4656-87f7-9a167fb10905 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:06.925131 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22318d62-0b69-4656-87f7-9a167fb10905 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:06.925417 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22318d62-0b69-4656-87f7-9a167fb10905 None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:06.926354 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22318d62-0b69-4656-87f7-9a167fb10905 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:06.926893 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22318d62-0b69-4656-87f7-9a167fb10905 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:06.928616 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22318d62-0b69-4656-87f7-9a167fb10905 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:06.942207 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 92/188] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Sep 25 04:22:06 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:06.942207 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c5b3ab3-2edf-43a8-a9e0-1aa73c9760a5 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:06.942207 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c5b3ab3-2edf-43a8-a9e0-1aa73c9760a5 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:06.942207 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c5b3ab3-2edf-43a8-a9e0-1aa73c9760a5 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:06.942583 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 93/189] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:22:06 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:22:06.943130 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c5b3ab3-2edf-43a8-a9e0-1aa73c9760a5 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:06.944660 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:07.007413 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82665ac3-9132-4833-a96c-600ae9a93a76 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:07.042479 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a23d0d1-5f92-4026-b145-ecf0777b17e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:07.043081 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a23d0d1-5f92-4026-b145-ecf0777b17e3 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:07.043362 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a23d0d1-5f92-4026-b145-ecf0777b17e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:07.043628 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a23d0d1-5f92-4026-b145-ecf0777b17e3 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:07.044628 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a23d0d1-5f92-4026-b145-ecf0777b17e3 None admin] RBAC: Adding query filter params (role.id=8f6cc574056c4278bc9778f14dc4b411, scope.project.id=bceb73400ebb4062be126679d2f16f6d, user.id=bc82e617c36e4fdfa13e159395d54e1c) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:07.044961 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a23d0d1-5f92-4026-b145-ecf0777b17e3 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:07.046391 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a23d0d1-5f92-4026-b145-ecf0777b17e3 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:07.052290 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 97/190] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Sep 25 04:22:06 2022] GET /identity/v3/role_assignments?user.id=bc82e617c36e4fdfa13e159395d54e1c&scope.project.id=bceb73400ebb4062be126679d2f16f6d&role.id=8f6cc574056c4278bc9778f14dc4b411 => generated 608 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:07.053168 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a23d0d1-5f92-4026-b145-ecf0777b17e3 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:07.057031 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c5b3ab3-2edf-43a8-a9e0-1aa73c9760a5 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:07.097177 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bffc9e1-a7aa-40fc-9c41-a09a2fd1ca09 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:07.097805 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bffc9e1-a7aa-40fc-9c41-a09a2fd1ca09 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:07.098087 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bffc9e1-a7aa-40fc-9c41-a09a2fd1ca09 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:07.098358 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bffc9e1-a7aa-40fc-9c41-a09a2fd1ca09 None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:07.099037 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bffc9e1-a7aa-40fc-9c41-a09a2fd1ca09 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:07.099341 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bffc9e1-a7aa-40fc-9c41-a09a2fd1ca09 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:07.100757 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bffc9e1-a7aa-40fc-9c41-a09a2fd1ca09 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:07.109653 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 94/191] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Sep 25 04:22:06 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:07.116356 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-185eb5f6-6f17-4aed-af01-0044d19958b0 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:07.116660 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-185eb5f6-6f17-4aed-af01-0044d19958b0 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:07.116937 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-185eb5f6-6f17-4aed-af01-0044d19958b0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:07.135735 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-185eb5f6-6f17-4aed-af01-0044d19958b0 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:22:07.137023 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-185eb5f6-6f17-4aed-af01-0044d19958b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:22:07.173006 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a23d0d1-5f92-4026-b145-ecf0777b17e3 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:07.184549 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:07.188614 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:07.211705 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 95/192] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:22:07 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 04:22:07.212552 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-185eb5f6-6f17-4aed-af01-0044d19958b0 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:07.220884 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f44241ee-2e25-45bc-9bd5-566c4f9a204b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:07.220884 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f44241ee-2e25-45bc-9bd5-566c4f9a204b None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:07.220884 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f44241ee-2e25-45bc-9bd5-566c4f9a204b None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:07.220884 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f44241ee-2e25-45bc-9bd5-566c4f9a204b None admin] PATH_INFO: `/v3/users/dbce0115f8df41a2964a6bf6677e25da` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:07.238995 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f44241ee-2e25-45bc-9bd5-566c4f9a204b None admin] RBAC: Authorizing `identity:get_user(user_id=dbce0115f8df41a2964a6bf6677e25da)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:07.240498 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f44241ee-2e25-45bc-9bd5-566c4f9a204b None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:07.242340 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 98/193] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Sep 25 04:22:07 2022] GET /identity/v3/users/dbce0115f8df41a2964a6bf6677e25da => generated 283 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:07.243107 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f44241ee-2e25-45bc-9bd5-566c4f9a204b None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:07.336760 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-185eb5f6-6f17-4aed-af01-0044d19958b0 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:07.381735 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b65d3f02-0964-495e-b8c4-c6bf5a69da08 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:07.382498 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b65d3f02-0964-495e-b8c4-c6bf5a69da08 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:07.382661 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b65d3f02-0964-495e-b8c4-c6bf5a69da08 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:07.382920 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b65d3f02-0964-495e-b8c4-c6bf5a69da08 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:07.384034 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b65d3f02-0964-495e-b8c4-c6bf5a69da08 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:07.385032 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f44241ee-2e25-45bc-9bd5-566c4f9a204b None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:07.385564 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b65d3f02-0964-495e-b8c4-c6bf5a69da08 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:07.387125 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 96/194] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:22:07 2022] GET /identity/v3/domains/default => generated 199 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:07.388035 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b65d3f02-0964-495e-b8c4-c6bf5a69da08 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:07.428083 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4e563de-9b57-47d4-9c5a-32db1abe8539 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:07.428652 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e563de-9b57-47d4-9c5a-32db1abe8539 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:07.428942 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e563de-9b57-47d4-9c5a-32db1abe8539 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:07.429210 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e563de-9b57-47d4-9c5a-32db1abe8539 None admin] PATH_INFO: `/v3/users/dbce0115f8df41a2964a6bf6677e25da` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:07.430326 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4e563de-9b57-47d4-9c5a-32db1abe8539 None admin] RBAC: Authorizing `identity:get_user(user_id=dbce0115f8df41a2964a6bf6677e25da)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:07.431757 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4e563de-9b57-47d4-9c5a-32db1abe8539 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:07.433352 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 99/195] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Sep 25 04:22:07 2022] GET /identity/v3/users/dbce0115f8df41a2964a6bf6677e25da => generated 283 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:07.437967 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4e563de-9b57-47d4-9c5a-32db1abe8539 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:07.446033 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:07.478863 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:07.503664 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b65d3f02-0964-495e-b8c4-c6bf5a69da08 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:07.538085 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac2ec7e3-329b-4225-afaf-1ec79d7c5351 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:07.538701 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac2ec7e3-329b-4225-afaf-1ec79d7c5351 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:07.538984 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac2ec7e3-329b-4225-afaf-1ec79d7c5351 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:07.539276 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac2ec7e3-329b-4225-afaf-1ec79d7c5351 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:07.544083 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4e563de-9b57-47d4-9c5a-32db1abe8539 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:07.577751 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-160e2bca-9de0-46c9-bc4f-5c64421d7756 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:07.578361 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-160e2bca-9de0-46c9-bc4f-5c64421d7756 None admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:07.578639 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-160e2bca-9de0-46c9-bc4f-5c64421d7756 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:07.578911 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-160e2bca-9de0-46c9-bc4f-5c64421d7756 None admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:07.580125 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-160e2bca-9de0-46c9-bc4f-5c64421d7756 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:07.581549 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-160e2bca-9de0-46c9-bc4f-5c64421d7756 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:07.602492 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 100/196] 10.222.0.25 () {60 vars in 1132 bytes} [Sun Sep 25 04:22:07 2022] POST /identity/v3/users => generated 297 bytes in 169 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:22:07.603552 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-160e2bca-9de0-46c9-bc4f-5c64421d7756 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:07.621065 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac2ec7e3-329b-4225-afaf-1ec79d7c5351 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:07.622991 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac2ec7e3-329b-4225-afaf-1ec79d7c5351 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:07.676778 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:07.708277 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:07.722134 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-160e2bca-9de0-46c9-bc4f-5c64421d7756 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:07.741960 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 97/197] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Sep 25 04:22:07 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 354 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 25 04:22:07.747572 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac2ec7e3-329b-4225-afaf-1ec79d7c5351 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:07.762173 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd42e4cf-c847-4b2c-aac1-883a1342c967 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:07.763077 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd42e4cf-c847-4b2c-aac1-883a1342c967 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:07.763353 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd42e4cf-c847-4b2c-aac1-883a1342c967 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:07.763620 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd42e4cf-c847-4b2c-aac1-883a1342c967 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:07.847683 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd42e4cf-c847-4b2c-aac1-883a1342c967 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:07.849501 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd42e4cf-c847-4b2c-aac1-883a1342c967 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:07.874820 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac2ec7e3-329b-4225-afaf-1ec79d7c5351 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:07.923929 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd6360c7-a3c9-498a-8d79-09c427361c65 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:07.924616 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd6360c7-a3c9-498a-8d79-09c427361c65 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:07.924927 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd6360c7-a3c9-498a-8d79-09c427361c65 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:07.925193 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd6360c7-a3c9-498a-8d79-09c427361c65 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:07.936508 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd6360c7-a3c9-498a-8d79-09c427361c65 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 04:22:07.942699 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-bd6360c7-a3c9-498a-8d79-09c427361c65 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 25 04:22:07.942699 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:22:07.942699 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:22:07.942699 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:22:07.942699 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:22:07.942699 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:22:07.942699 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:22:07.942699 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:22:07.942699 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:22:07.942699 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:22:07.942699 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:22:07.942699 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:22:07.942699 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 04:22:07.942699 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 04:22:07.942699 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 04:22:07.942699 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 04:22:07.943617 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 04:22:07.943617 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 04:22:07.943617 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 04:22:07.943617 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 04:22:07.943617 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:22:07.943617 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:22:07.943617 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:22:07.943617 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:22:07.943617 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:22:07.943617 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:22:07.943617 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:22:07.943617 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:22:07.943617 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:22:07.943617 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:22:07.943617 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:22:07.943617 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:22:07.945060 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:22:07.945060 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:22:07.945060 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:22:07.945060 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:22:07.945060 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 04:22:07.945060 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 04:22:07.945060 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 04:22:07.945060 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 04:22:07.945060 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 04:22:07.945060 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 04:22:07.945060 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 25 04:22:07.945060 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:22:07.945060 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 98/198] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Sep 25 04:22:07 2022] GET /identity/v3/roles/reader => generated 84 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:22:07.945060 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd6360c7-a3c9-498a-8d79-09c427361c65 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:07.959943 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:07.977676 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:07.996030 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 101/199] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Sep 25 04:22:07 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 393 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 25 04:22:07.996739 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd42e4cf-c847-4b2c-aac1-883a1342c967 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:08.055316 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd6360c7-a3c9-498a-8d79-09c427361c65 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:08.091540 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73f57b32-8abc-4722-a257-fe536651eee3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:08.092356 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73f57b32-8abc-4722-a257-fe536651eee3 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:08.092656 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73f57b32-8abc-4722-a257-fe536651eee3 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:08.093016 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73f57b32-8abc-4722-a257-fe536651eee3 None admin] PATH_INFO: `/v3/projects/12400184cf3844998e5e6d8078b8cda4` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:08.094372 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73f57b32-8abc-4722-a257-fe536651eee3 None admin] RBAC: Authorizing `identity:get_project(project_id=12400184cf3844998e5e6d8078b8cda4)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:08.096522 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73f57b32-8abc-4722-a257-fe536651eee3 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:08.098299 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 99/200] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Sep 25 04:22:07 2022] GET /identity/v3/projects/12400184cf3844998e5e6d8078b8cda4 => generated 302 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:08.104070 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73f57b32-8abc-4722-a257-fe536651eee3 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:08.133693 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd42e4cf-c847-4b2c-aac1-883a1342c967 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:08.169781 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-090b339b-3eb5-4b07-aca6-0e4ef633ab3b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:08.170530 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-090b339b-3eb5-4b07-aca6-0e4ef633ab3b None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:08.170848 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-090b339b-3eb5-4b07-aca6-0e4ef633ab3b None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:08.171216 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-090b339b-3eb5-4b07-aca6-0e4ef633ab3b None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:08.171951 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-090b339b-3eb5-4b07-aca6-0e4ef633ab3b None admin] RBAC: Adding query filter params (name=reader) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:08.172423 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-090b339b-3eb5-4b07-aca6-0e4ef633ab3b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:08.174592 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-090b339b-3eb5-4b07-aca6-0e4ef633ab3b None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:08.183692 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 102/201] 10.222.0.25 () {56 vars in 1101 bytes} [Sun Sep 25 04:22:07 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:08.184764 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-090b339b-3eb5-4b07-aca6-0e4ef633ab3b None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:08.206196 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:08.244740 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73f57b32-8abc-4722-a257-fe536651eee3 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:08.249203 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:08.290234 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b72bfae7-5787-4196-b1e3-f033493a685c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:08.291316 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b72bfae7-5787-4196-b1e3-f033493a685c None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:08.291813 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b72bfae7-5787-4196-b1e3-f033493a685c None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:08.292452 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b72bfae7-5787-4196-b1e3-f033493a685c None admin] PATH_INFO: `/v3/projects/12400184cf3844998e5e6d8078b8cda4` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:08.294206 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b72bfae7-5787-4196-b1e3-f033493a685c None admin] RBAC: Authorizing `identity:get_project(project_id=12400184cf3844998e5e6d8078b8cda4)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:08.295984 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b72bfae7-5787-4196-b1e3-f033493a685c None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:08.298144 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 100/202] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Sep 25 04:22:08 2022] GET /identity/v3/projects/12400184cf3844998e5e6d8078b8cda4 => generated 302 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:08.299643 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b72bfae7-5787-4196-b1e3-f033493a685c None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:08.316449 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-090b339b-3eb5-4b07-aca6-0e4ef633ab3b None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:08.357745 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5b70c48-4cb4-4a86-a068-2193c9e392d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:08.358646 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5b70c48-4cb4-4a86-a068-2193c9e392d9 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:08.359322 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5b70c48-4cb4-4a86-a068-2193c9e392d9 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:08.359747 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5b70c48-4cb4-4a86-a068-2193c9e392d9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:08.360617 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5b70c48-4cb4-4a86-a068-2193c9e392d9 None admin] RBAC: Adding query filter params (role.id=c51fe7c5ad804040879acfb54269455f, scope.project.id=12400184cf3844998e5e6d8078b8cda4, user.id=dbce0115f8df41a2964a6bf6677e25da) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:08.361633 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5b70c48-4cb4-4a86-a068-2193c9e392d9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:08.363329 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5b70c48-4cb4-4a86-a068-2193c9e392d9 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:08.370203 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 103/203] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Sep 25 04:22:08 2022] GET /identity/v3/role_assignments?user.id=dbce0115f8df41a2964a6bf6677e25da&scope.project.id=12400184cf3844998e5e6d8078b8cda4&role.id=c51fe7c5ad804040879acfb54269455f => generated 259 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:08.371141 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5b70c48-4cb4-4a86-a068-2193c9e392d9 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:08.460002 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b72bfae7-5787-4196-b1e3-f033493a685c None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:08.536804 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83da504e-88c4-4625-8208-7f74a5bc3aef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:08.537445 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83da504e-88c4-4625-8208-7f74a5bc3aef None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:08.537734 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83da504e-88c4-4625-8208-7f74a5bc3aef None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:08.538009 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83da504e-88c4-4625-8208-7f74a5bc3aef None admin] PATH_INFO: `/v3/users/f525f4e8fcf2462b9f0df8ef2197b752` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:08.539367 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83da504e-88c4-4625-8208-7f74a5bc3aef None admin] RBAC: Authorizing `identity:get_user(user_id=f525f4e8fcf2462b9f0df8ef2197b752)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:08.540933 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83da504e-88c4-4625-8208-7f74a5bc3aef None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:08.542669 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 101/204] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Sep 25 04:22:08 2022] GET /identity/v3/users/f525f4e8fcf2462b9f0df8ef2197b752 => generated 289 bytes in 244 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:08.548518 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83da504e-88c4-4625-8208-7f74a5bc3aef None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:08.578928 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:08.601677 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5b70c48-4cb4-4a86-a068-2193c9e392d9 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:08.624364 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:08.648125 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4056111e-582a-4363-9eb7-ffbe3d3dcb69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:08.649178 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4056111e-582a-4363-9eb7-ffbe3d3dcb69 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:08.649670 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4056111e-582a-4363-9eb7-ffbe3d3dcb69 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:08.650202 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4056111e-582a-4363-9eb7-ffbe3d3dcb69 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:08.651242 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4056111e-582a-4363-9eb7-ffbe3d3dcb69 None admin] RBAC: Adding query filter params (role.id=02a8d263519b4ff1b511a8577293126c, scope.project.id=12400184cf3844998e5e6d8078b8cda4, user.id=dbce0115f8df41a2964a6bf6677e25da) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:08.651806 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4056111e-582a-4363-9eb7-ffbe3d3dcb69 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:08.653648 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4056111e-582a-4363-9eb7-ffbe3d3dcb69 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:08.661304 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 104/205] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Sep 25 04:22:08 2022] GET /identity/v3/role_assignments?user.id=dbce0115f8df41a2964a6bf6677e25da&scope.project.id=12400184cf3844998e5e6d8078b8cda4&role.id=02a8d263519b4ff1b511a8577293126c => generated 259 bytes in 290 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:08.695973 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83da504e-88c4-4625-8208-7f74a5bc3aef None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:08.737764 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd180039-cf8f-4b0b-bde5-f3512ae34f6d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:08.738407 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd180039-cf8f-4b0b-bde5-f3512ae34f6d None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:08.738688 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd180039-cf8f-4b0b-bde5-f3512ae34f6d None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:08.738961 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd180039-cf8f-4b0b-bde5-f3512ae34f6d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:08.813520 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd180039-cf8f-4b0b-bde5-f3512ae34f6d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:08.815201 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd180039-cf8f-4b0b-bde5-f3512ae34f6d None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:08.830802 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:08.923839 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 102/206] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Sep 25 04:22:08 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 375 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 25 04:22:08.929509 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4056111e-582a-4363-9eb7-ffbe3d3dcb69 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:09.062676 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4056111e-582a-4363-9eb7-ffbe3d3dcb69 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:09.100044 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96f5a9fe-ccf6-469f-a6b1-3d535dc97be8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:09.100602 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96f5a9fe-ccf6-469f-a6b1-3d535dc97be8 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:09.100910 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96f5a9fe-ccf6-469f-a6b1-3d535dc97be8 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:09.101184 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96f5a9fe-ccf6-469f-a6b1-3d535dc97be8 None admin] PATH_INFO: `/v3/projects/bceb73400ebb4062be126679d2f16f6d` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:09.102413 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96f5a9fe-ccf6-469f-a6b1-3d535dc97be8 None admin] RBAC: Authorizing `identity:get_project(project_id=bceb73400ebb4062be126679d2f16f6d)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:09.103892 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96f5a9fe-ccf6-469f-a6b1-3d535dc97be8 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:09.105553 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 105/207] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Sep 25 04:22:08 2022] GET /identity/v3/projects/bceb73400ebb4062be126679d2f16f6d => generated 298 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:09.112129 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd180039-cf8f-4b0b-bde5-f3512ae34f6d None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:09.255867 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd180039-cf8f-4b0b-bde5-f3512ae34f6d None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:09.272519 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:09.301683 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21da89b6-14f9-48da-8a91-5e2aafa2e3c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:09.302300 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21da89b6-14f9-48da-8a91-5e2aafa2e3c0 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:09.302588 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21da89b6-14f9-48da-8a91-5e2aafa2e3c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:09.302859 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21da89b6-14f9-48da-8a91-5e2aafa2e3c0 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:09.303587 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21da89b6-14f9-48da-8a91-5e2aafa2e3c0 None admin] RBAC: Adding query filter params (role.id=0e4fa3b3402d463da90ddfcd17be7c17, scope.project.id=bceb73400ebb4062be126679d2f16f6d, user.id=f525f4e8fcf2462b9f0df8ef2197b752) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:09.303893 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21da89b6-14f9-48da-8a91-5e2aafa2e3c0 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:09.305423 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21da89b6-14f9-48da-8a91-5e2aafa2e3c0 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:09.311785 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 103/208] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Sep 25 04:22:09 2022] GET /identity/v3/role_assignments?user.id=f525f4e8fcf2462b9f0df8ef2197b752&scope.project.id=bceb73400ebb4062be126679d2f16f6d&role.id=0e4fa3b3402d463da90ddfcd17be7c17 => generated 608 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:10.023801 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9dd65d0-4821-464b-9aca-99cbab2adad1 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:10.024313 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9dd65d0-4821-464b-9aca-99cbab2adad1 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:10.024419 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9dd65d0-4821-464b-9aca-99cbab2adad1 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:10.025758 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 106/209] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:22:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:22:10.031128 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab6c5576-0208-429d-a876-e519df9bc59d None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:10.031540 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab6c5576-0208-429d-a876-e519df9bc59d None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:10.031713 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab6c5576-0208-429d-a876-e519df9bc59d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:10.053629 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-ab6c5576-0208-429d-a876-e519df9bc59d None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:22:10.054819 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-ab6c5576-0208-429d-a876-e519df9bc59d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:22:10.124992 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 104/210] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:22:10 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 25 04:22:10.280659 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43bbc724-99f4-4bfb-a296-e95c190fc98b None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:10.280958 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43bbc724-99f4-4bfb-a296-e95c190fc98b None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:10.281144 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43bbc724-99f4-4bfb-a296-e95c190fc98b None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:10.282986 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 107/211] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:22:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:22:10.288330 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-779e43ad-f2e5-4c45-bb95-a184a30fbbb9 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:10.288880 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-779e43ad-f2e5-4c45-bb95-a184a30fbbb9 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:10.289361 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-779e43ad-f2e5-4c45-bb95-a184a30fbbb9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:10.311268 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-779e43ad-f2e5-4c45-bb95-a184a30fbbb9 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:22:10.312460 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-779e43ad-f2e5-4c45-bb95-a184a30fbbb9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:22:10.381894 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 105/212] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:22:10 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 04:22:10.400669 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43bbc724-99f4-4bfb-a296-e95c190fc98b None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:10.453710 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:10.508412 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43bbc724-99f4-4bfb-a296-e95c190fc98b None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:10.543297 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76b30171-8321-48d1-8cad-62b92e7de8ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:10.543870 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76b30171-8321-48d1-8cad-62b92e7de8ad None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:10.544147 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76b30171-8321-48d1-8cad-62b92e7de8ad None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:10.544421 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76b30171-8321-48d1-8cad-62b92e7de8ad None admin] PATH_INFO: `/v3/roles/member` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:10.555386 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76b30171-8321-48d1-8cad-62b92e7de8ad None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 04:22:10.560731 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-76b30171-8321-48d1-8cad-62b92e7de8ad None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 25 04:22:10.560731 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:22:10.560731 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:22:10.560731 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:22:10.560731 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:22:10.560731 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:22:10.560731 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:22:10.560731 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:22:10.560731 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:22:10.560731 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:22:10.560731 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:22:10.560731 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:22:10.560731 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 04:22:10.560731 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 04:22:10.560731 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 04:22:10.560731 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 04:22:10.561635 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 04:22:10.561635 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 04:22:10.561635 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 04:22:10.561635 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 04:22:10.561635 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:22:10.561635 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:22:10.561635 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:22:10.561635 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:22:10.561635 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:22:10.561635 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:22:10.561635 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:22:10.561635 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:22:10.561635 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:22:10.561635 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:22:10.561635 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:22:10.561635 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:22:10.562598 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:22:10.562598 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:22:10.562598 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:22:10.562598 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:22:10.562598 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 04:22:10.562598 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 04:22:10.562598 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 04:22:10.562598 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 04:22:10.562598 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 04:22:10.562598 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 04:22:10.562598 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 25 04:22:10.562598 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:22:10.562598 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 108/213] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Sep 25 04:22:10 2022] GET /identity/v3/roles/member => generated 84 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:22:10.567879 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-779e43ad-f2e5-4c45-bb95-a184a30fbbb9 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:10.604515 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:10.677421 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76b30171-8321-48d1-8cad-62b92e7de8ad None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:10.678728 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-779e43ad-f2e5-4c45-bb95-a184a30fbbb9 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:10.715090 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e3cf8d1-b34a-40ad-ad7a-f3a52987b1a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:10.715707 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e3cf8d1-b34a-40ad-ad7a-f3a52987b1a9 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:10.715992 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e3cf8d1-b34a-40ad-ad7a-f3a52987b1a9 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:10.716263 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e3cf8d1-b34a-40ad-ad7a-f3a52987b1a9 None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:10.716961 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e3cf8d1-b34a-40ad-ad7a-f3a52987b1a9 None admin] RBAC: Adding query filter params (name=member) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:10.717265 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e3cf8d1-b34a-40ad-ad7a-f3a52987b1a9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:10.718650 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e3cf8d1-b34a-40ad-ad7a-f3a52987b1a9 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:10.727828 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 106/214] 10.222.0.25 () {56 vars in 1101 bytes} [Sun Sep 25 04:22:10 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:10.733471 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e3cf8d1-b34a-40ad-ad7a-f3a52987b1a9 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:10.801888 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76b30171-8321-48d1-8cad-62b92e7de8ad None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:10.814246 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:10.836713 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:10.840531 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e3cf8d1-b34a-40ad-ad7a-f3a52987b1a9 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:10.849944 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e59f5bc2-d275-4633-ac05-0564dce844cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:10.850591 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e59f5bc2-d275-4633-ac05-0564dce844cc None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:10.850868 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e59f5bc2-d275-4633-ac05-0564dce844cc None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:10.851138 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e59f5bc2-d275-4633-ac05-0564dce844cc None admin] PATH_INFO: `/v3/domains/default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:10.853981 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e59f5bc2-d275-4633-ac05-0564dce844cc None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:10.855431 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e59f5bc2-d275-4633-ac05-0564dce844cc None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:10.857111 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 109/215] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:22:10 2022] GET /identity/v3/domains/default => generated 199 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:22:10.863640 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e59f5bc2-d275-4633-ac05-0564dce844cc None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:10.887657 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-198c5a16-d100-4173-8bc3-db32b8855394 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:10.888477 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-198c5a16-d100-4173-8bc3-db32b8855394 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:10.888890 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-198c5a16-d100-4173-8bc3-db32b8855394 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:10.889263 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-198c5a16-d100-4173-8bc3-db32b8855394 None admin] PATH_INFO: `/v3/users/4f5a0c33729d4e7d9813103a3a5acef7` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:10.905585 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-198c5a16-d100-4173-8bc3-db32b8855394 None admin] RBAC: Authorizing `identity:get_user(user_id=4f5a0c33729d4e7d9813103a3a5acef7)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:10.907273 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-198c5a16-d100-4173-8bc3-db32b8855394 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:10.910002 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 107/216] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Sep 25 04:22:10 2022] GET /identity/v3/users/4f5a0c33729d4e7d9813103a3a5acef7 => generated 297 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:22:10.915829 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-198c5a16-d100-4173-8bc3-db32b8855394 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:10.995346 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e59f5bc2-d275-4633-ac05-0564dce844cc None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:11.023265 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-198c5a16-d100-4173-8bc3-db32b8855394 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:11.033263 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22b7a960-598e-47c0-ba47-9d342d84c30e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:11.033928 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22b7a960-598e-47c0-ba47-9d342d84c30e None admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:11.034149 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22b7a960-598e-47c0-ba47-9d342d84c30e None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:11.034586 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22b7a960-598e-47c0-ba47-9d342d84c30e None admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:11.035242 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22b7a960-598e-47c0-ba47-9d342d84c30e None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:11.036838 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22b7a960-598e-47c0-ba47-9d342d84c30e None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:11.067987 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 110/217] 10.222.0.25 () {60 vars in 1132 bytes} [Sun Sep 25 04:22:10 2022] POST /identity/v3/users => generated 297 bytes in 205 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:22:11.069431 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-899fdc07-dd74-4c6e-bb19-38518620eded None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:11.069724 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-899fdc07-dd74-4c6e-bb19-38518620eded None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:11.070050 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-899fdc07-dd74-4c6e-bb19-38518620eded None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:11.071107 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e32f511f-a2fd-42bd-8790-e90531946e58 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:11.071741 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e32f511f-a2fd-42bd-8790-e90531946e58 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:11.072068 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e32f511f-a2fd-42bd-8790-e90531946e58 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:11.072338 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e32f511f-a2fd-42bd-8790-e90531946e58 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:11.074475 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 111/218] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:22:11 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:22:11.081698 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-122248b7-594e-4e1f-9dc8-56afe34a4049 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:11.082270 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-122248b7-594e-4e1f-9dc8-56afe34a4049 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:11.082781 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-122248b7-594e-4e1f-9dc8-56afe34a4049 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:11.104617 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-122248b7-594e-4e1f-9dc8-56afe34a4049 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:22:11.106251 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-122248b7-594e-4e1f-9dc8-56afe34a4049 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:22:11.115625 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:11.143441 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:11.186164 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e32f511f-a2fd-42bd-8790-e90531946e58 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:11.188445 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e32f511f-a2fd-42bd-8790-e90531946e58 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:11.198797 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 112/219] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:22:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 25 04:22:11.243439 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ef27ab1-b2de-487f-a300-570dd253fcfd None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:11.244068 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ef27ab1-b2de-487f-a300-570dd253fcfd None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:11.244534 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ef27ab1-b2de-487f-a300-570dd253fcfd None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:11.246762 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 113/220] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:22:11 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:22:11.252265 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eed91491-875e-4587-a761-cab5c5b382d6 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:11.252550 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eed91491-875e-4587-a761-cab5c5b382d6 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:11.252855 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eed91491-875e-4587-a761-cab5c5b382d6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:11.273051 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-eed91491-875e-4587-a761-cab5c5b382d6 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:22:11.274247 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-eed91491-875e-4587-a761-cab5c5b382d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:22:11.307420 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 108/221] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Sep 25 04:22:10 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 392 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 25 04:22:11.312960 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e32f511f-a2fd-42bd-8790-e90531946e58 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:11.358953 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 114/222] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:22:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 04:22:11.379192 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:11.427624 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e32f511f-a2fd-42bd-8790-e90531946e58 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:11.463106 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d831b6e3-ef49-4b4b-ae23-abb9093a6da8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:11.463755 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d831b6e3-ef49-4b4b-ae23-abb9093a6da8 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:11.464039 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d831b6e3-ef49-4b4b-ae23-abb9093a6da8 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:11.464314 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d831b6e3-ef49-4b4b-ae23-abb9093a6da8 None admin] PATH_INFO: `/v3/projects/12400184cf3844998e5e6d8078b8cda4` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:11.467765 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d831b6e3-ef49-4b4b-ae23-abb9093a6da8 None admin] RBAC: Authorizing `identity:get_project(project_id=12400184cf3844998e5e6d8078b8cda4)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:11.467765 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d831b6e3-ef49-4b4b-ae23-abb9093a6da8 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:11.468562 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 109/223] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Sep 25 04:22:11 2022] GET /identity/v3/projects/12400184cf3844998e5e6d8078b8cda4 => generated 302 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:22:11.478757 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eed91491-875e-4587-a761-cab5c5b382d6 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:11.500920 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d831b6e3-ef49-4b4b-ae23-abb9093a6da8 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:11.593701 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:11.604988 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eed91491-875e-4587-a761-cab5c5b382d6 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:11.635811 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d831b6e3-ef49-4b4b-ae23-abb9093a6da8 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:11.652568 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc7568cc-5f14-4722-bdf2-dc877fb582e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:11.653713 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc7568cc-5f14-4722-bdf2-dc877fb582e7 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:11.654222 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc7568cc-5f14-4722-bdf2-dc877fb582e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:11.654991 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc7568cc-5f14-4722-bdf2-dc877fb582e7 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:11.655929 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc7568cc-5f14-4722-bdf2-dc877fb582e7 None admin] RBAC: Adding query filter params (role.id=8f6cc574056c4278bc9778f14dc4b411, scope.project.id=12400184cf3844998e5e6d8078b8cda4, user.id=4f5a0c33729d4e7d9813103a3a5acef7) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:11.656446 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc7568cc-5f14-4722-bdf2-dc877fb582e7 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:11.659100 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc7568cc-5f14-4722-bdf2-dc877fb582e7 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:11.661868 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:11.666727 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 115/224] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Sep 25 04:22:11 2022] GET /identity/v3/role_assignments?user.id=4f5a0c33729d4e7d9813103a3a5acef7&scope.project.id=12400184cf3844998e5e6d8078b8cda4&role.id=8f6cc574056c4278bc9778f14dc4b411 => generated 259 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:11.670617 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc7568cc-5f14-4722-bdf2-dc877fb582e7 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:11.678833 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63935240-aa1f-459d-96ef-cb5f3051df00 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:11.680553 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63935240-aa1f-459d-96ef-cb5f3051df00 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:11.681099 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63935240-aa1f-459d-96ef-cb5f3051df00 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:11.681597 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63935240-aa1f-459d-96ef-cb5f3051df00 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:11.700441 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63935240-aa1f-459d-96ef-cb5f3051df00 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 04:22:11.707903 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-63935240-aa1f-459d-96ef-cb5f3051df00 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 04:22:11.707903 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:22:11.707903 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:22:11.707903 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:22:11.707903 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:22:11.707903 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:22:11.707903 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:22:11.707903 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:22:11.707903 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:22:11.707903 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:22:11.707903 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:22:11.707903 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:22:11.707903 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 04:22:11.707903 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 04:22:11.707903 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 04:22:11.707903 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 04:22:11.708929 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 04:22:11.708929 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 04:22:11.708929 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 04:22:11.708929 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 04:22:11.708929 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:22:11.708929 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:22:11.708929 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:22:11.708929 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:22:11.708929 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:22:11.708929 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:22:11.708929 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:22:11.708929 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:22:11.708929 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:22:11.708929 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:22:11.708929 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:22:11.708929 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:22:11.709747 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:22:11.709747 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:22:11.709747 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:22:11.709747 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:22:11.709747 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 04:22:11.709747 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 04:22:11.709747 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 04:22:11.709747 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 04:22:11.709747 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 04:22:11.709747 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 04:22:11.709747 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 04:22:11.709747 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:22:11.710371 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 110/225] 10.222.0.25 () {56 vars in 1096 bytes} [Sun Sep 25 04:22:11 2022] GET /identity/v3/roles/admin => generated 83 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:22:11.717480 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63935240-aa1f-459d-96ef-cb5f3051df00 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:11.840157 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc7568cc-5f14-4722-bdf2-dc877fb582e7 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:11.852624 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63935240-aa1f-459d-96ef-cb5f3051df00 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:11.895667 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7903c092-4602-44ef-b6ac-3fd50af24e88 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:11.896789 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7903c092-4602-44ef-b6ac-3fd50af24e88 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:11.897294 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7903c092-4602-44ef-b6ac-3fd50af24e88 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:11.897769 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7903c092-4602-44ef-b6ac-3fd50af24e88 None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:11.898762 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7903c092-4602-44ef-b6ac-3fd50af24e88 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:11.899284 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7903c092-4602-44ef-b6ac-3fd50af24e88 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:11.901030 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7903c092-4602-44ef-b6ac-3fd50af24e88 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:11.913085 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-755a9adc-cbff-456f-896f-91f809e2b81e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:11.914133 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-755a9adc-cbff-456f-896f-91f809e2b81e None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:11.914615 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-755a9adc-cbff-456f-896f-91f809e2b81e None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:11.915083 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-755a9adc-cbff-456f-896f-91f809e2b81e None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:11.919544 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 111/226] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Sep 25 04:22:11 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:11.925748 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7903c092-4602-44ef-b6ac-3fd50af24e88 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:11.934960 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-755a9adc-cbff-456f-896f-91f809e2b81e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 04:22:11.941248 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-755a9adc-cbff-456f-896f-91f809e2b81e None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 25 04:22:11.941248 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:22:11.941248 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:22:11.941248 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:22:11.941248 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:22:11.941248 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:22:11.941248 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:22:11.941248 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:22:11.941248 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:22:11.941248 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:22:11.941248 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:22:11.941248 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:22:11.941248 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 04:22:11.941248 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 04:22:11.941248 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 04:22:11.941248 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 04:22:11.942203 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 04:22:11.942203 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 04:22:11.942203 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 04:22:11.942203 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 04:22:11.942203 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:22:11.942203 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:22:11.942203 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:22:11.942203 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:22:11.942203 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:22:11.942203 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:22:11.942203 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:22:11.942203 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:22:11.942203 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:22:11.942203 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:22:11.942203 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:22:11.942203 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:22:11.943001 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:22:11.943001 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:22:11.943001 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:22:11.943001 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:22:11.943001 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 04:22:11.943001 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 04:22:11.943001 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 04:22:11.943001 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 04:22:11.943001 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 04:22:11.943001 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 04:22:11.943001 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 25 04:22:11.943001 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:22:11.943001 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 116/227] 10.222.0.25 () {56 vars in 1114 bytes} [Sun Sep 25 04:22:11 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 272 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 04:22:11.949455 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-755a9adc-cbff-456f-896f-91f809e2b81e None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:11.980038 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:11.997081 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:12.096670 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7903c092-4602-44ef-b6ac-3fd50af24e88 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:12.138660 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-755a9adc-cbff-456f-896f-91f809e2b81e None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:12.156567 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28f8ca9a-d77f-429a-9b25-4ac9e333280e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:12.157163 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28f8ca9a-d77f-429a-9b25-4ac9e333280e None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:12.157641 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28f8ca9a-d77f-429a-9b25-4ac9e333280e None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:12.157917 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28f8ca9a-d77f-429a-9b25-4ac9e333280e None admin] PATH_INFO: `/v3/users/dbce0115f8df41a2964a6bf6677e25da` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:12.160490 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28f8ca9a-d77f-429a-9b25-4ac9e333280e None admin] RBAC: Authorizing `identity:get_user(user_id=dbce0115f8df41a2964a6bf6677e25da)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:12.161984 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28f8ca9a-d77f-429a-9b25-4ac9e333280e None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:12.163907 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 112/228] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Sep 25 04:22:11 2022] GET /identity/v3/users/dbce0115f8df41a2964a6bf6677e25da => generated 283 bytes in 238 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:12.169263 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28f8ca9a-d77f-429a-9b25-4ac9e333280e None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:12.205870 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5053137-2a48-434d-8ad7-9960ed82018d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:12.206488 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5053137-2a48-434d-8ad7-9960ed82018d None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:12.206761 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5053137-2a48-434d-8ad7-9960ed82018d None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:12.207026 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5053137-2a48-434d-8ad7-9960ed82018d None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:12.207693 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5053137-2a48-434d-8ad7-9960ed82018d None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:12.207995 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5053137-2a48-434d-8ad7-9960ed82018d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:12.209638 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5053137-2a48-434d-8ad7-9960ed82018d None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:12.219275 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 117/229] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Sep 25 04:22:11 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 270 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:12.225483 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5053137-2a48-434d-8ad7-9960ed82018d None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:12.295456 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28f8ca9a-d77f-429a-9b25-4ac9e333280e None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:12.323126 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:12.339041 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be843076-5786-4102-ad48-b12c3fc9b329 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:12.340163 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be843076-5786-4102-ad48-b12c3fc9b329 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:12.341616 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be843076-5786-4102-ad48-b12c3fc9b329 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:12.342265 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be843076-5786-4102-ad48-b12c3fc9b329 None admin] PATH_INFO: `/v3/projects/12400184cf3844998e5e6d8078b8cda4` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:12.344917 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be843076-5786-4102-ad48-b12c3fc9b329 None admin] RBAC: Authorizing `identity:get_project(project_id=12400184cf3844998e5e6d8078b8cda4)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:12.347494 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be843076-5786-4102-ad48-b12c3fc9b329 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:12.351146 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 113/230] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Sep 25 04:22:12 2022] GET /identity/v3/projects/12400184cf3844998e5e6d8078b8cda4 => generated 302 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:22:12.357598 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be843076-5786-4102-ad48-b12c3fc9b329 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:12.402405 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:12.445557 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5053137-2a48-434d-8ad7-9960ed82018d None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:12.513781 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a19f504-88f1-45b2-ac03-4ede14091981 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:12.514839 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a19f504-88f1-45b2-ac03-4ede14091981 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:12.515123 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a19f504-88f1-45b2-ac03-4ede14091981 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:12.515395 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a19f504-88f1-45b2-ac03-4ede14091981 None admin] PATH_INFO: `/v3/users/dbce0115f8df41a2964a6bf6677e25da` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:12.517984 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a19f504-88f1-45b2-ac03-4ede14091981 None admin] RBAC: Authorizing `identity:get_user(user_id=dbce0115f8df41a2964a6bf6677e25da)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:12.519463 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a19f504-88f1-45b2-ac03-4ede14091981 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:12.525880 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 118/231] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Sep 25 04:22:12 2022] GET /identity/v3/users/dbce0115f8df41a2964a6bf6677e25da => generated 283 bytes in 301 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:12.531181 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a19f504-88f1-45b2-ac03-4ede14091981 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:12.556802 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be843076-5786-4102-ad48-b12c3fc9b329 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:12.631239 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54415933-708b-458d-bd0e-04cae6e65d38 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:12.631921 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54415933-708b-458d-bd0e-04cae6e65d38 None admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:12.632202 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54415933-708b-458d-bd0e-04cae6e65d38 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:12.632476 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54415933-708b-458d-bd0e-04cae6e65d38 None admin] PATH_INFO: `/v3/projects/12400184cf3844998e5e6d8078b8cda4/users/dbce0115f8df41a2964a6bf6677e25da/roles/c51fe7c5ad804040879acfb54269455f` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:12.636713 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54415933-708b-458d-bd0e-04cae6e65d38 None admin] RBAC: Authorizing `identity:create_grant(project_id=12400184cf3844998e5e6d8078b8cda4, user_id=dbce0115f8df41a2964a6bf6677e25da, role_id=c51fe7c5ad804040879acfb54269455f)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:12.638174 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54415933-708b-458d-bd0e-04cae6e65d38 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:12.663875 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 114/232] 10.222.0.25 () {58 vars in 1439 bytes} [Sun Sep 25 04:22:12 2022] PUT /identity/v3/projects/12400184cf3844998e5e6d8078b8cda4/users/dbce0115f8df41a2964a6bf6677e25da/roles/c51fe7c5ad804040879acfb54269455f => generated 0 bytes in 306 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:22:12.704960 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a19f504-88f1-45b2-ac03-4ede14091981 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:12.742202 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0e25a5f-99a7-48eb-822f-07425ceda315 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:12.742862 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e25a5f-99a7-48eb-822f-07425ceda315 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:12.743142 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e25a5f-99a7-48eb-822f-07425ceda315 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:12.743418 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e25a5f-99a7-48eb-822f-07425ceda315 None admin] PATH_INFO: `/v3/projects/12400184cf3844998e5e6d8078b8cda4` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:12.744579 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0e25a5f-99a7-48eb-822f-07425ceda315 None admin] RBAC: Authorizing `identity:get_project(project_id=12400184cf3844998e5e6d8078b8cda4)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:12.746156 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0e25a5f-99a7-48eb-822f-07425ceda315 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:12.748077 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 119/233] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Sep 25 04:22:12 2022] GET /identity/v3/projects/12400184cf3844998e5e6d8078b8cda4 => generated 302 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:12.753838 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54415933-708b-458d-bd0e-04cae6e65d38 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:12.820525 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:12.899787 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54415933-708b-458d-bd0e-04cae6e65d38 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:12.956145 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b47a8643-b7a7-4695-8f98-6c74f2a9132b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:12.956776 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47a8643-b7a7-4695-8f98-6c74f2a9132b None admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:12.957063 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47a8643-b7a7-4695-8f98-6c74f2a9132b None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:12.957339 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47a8643-b7a7-4695-8f98-6c74f2a9132b None admin] PATH_INFO: `/v3/projects/12400184cf3844998e5e6d8078b8cda4/users/dbce0115f8df41a2964a6bf6677e25da/roles/02a8d263519b4ff1b511a8577293126c` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:12.960619 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b47a8643-b7a7-4695-8f98-6c74f2a9132b None admin] RBAC: Authorizing `identity:create_grant(project_id=12400184cf3844998e5e6d8078b8cda4, user_id=dbce0115f8df41a2964a6bf6677e25da, role_id=02a8d263519b4ff1b511a8577293126c)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:12.962129 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b47a8643-b7a7-4695-8f98-6c74f2a9132b None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:12.981801 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 115/234] 10.222.0.25 () {58 vars in 1439 bytes} [Sun Sep 25 04:22:12 2022] PUT /identity/v3/projects/12400184cf3844998e5e6d8078b8cda4/users/dbce0115f8df41a2964a6bf6677e25da/roles/02a8d263519b4ff1b511a8577293126c => generated 0 bytes in 228 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:22:13.992329 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61104a96-3d4b-4c1b-a451-dcefbad9c1e0 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:13.992923 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61104a96-3d4b-4c1b-a451-dcefbad9c1e0 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:13.992923 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61104a96-3d4b-4c1b-a451-dcefbad9c1e0 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:13.994402 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 120/235] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:22:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:22:14.000264 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c30b269a-1229-469b-8b87-9bc66f3f3f84 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:14.000801 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c30b269a-1229-469b-8b87-9bc66f3f3f84 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:14.000907 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c30b269a-1229-469b-8b87-9bc66f3f3f84 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:14.021479 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-c30b269a-1229-469b-8b87-9bc66f3f3f84 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:22:14.022686 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-c30b269a-1229-469b-8b87-9bc66f3f3f84 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:22:14.100009 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1804c1da-e5cb-4f0a-8b08-0430cf4464af None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:14.100352 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1804c1da-e5cb-4f0a-8b08-0430cf4464af None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:14.100563 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1804c1da-e5cb-4f0a-8b08-0430cf4464af None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:14.102078 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 121/236] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:22:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:22:14.107117 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b875a04-3e87-45fa-84dc-b69b8073919e None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:14.107618 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b875a04-3e87-45fa-84dc-b69b8073919e None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:14.108088 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b875a04-3e87-45fa-84dc-b69b8073919e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:14.124667 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 116/237] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:22:13 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 04:22:14.145768 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-7b875a04-3e87-45fa-84dc-b69b8073919e None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:22:14.147302 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-7b875a04-3e87-45fa-84dc-b69b8073919e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:22:14.163083 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:14.244913 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 122/238] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:22:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 25 04:22:14.354414 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c084f5a-745d-4076-b440-f6f46cd803d6 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:14.354756 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c084f5a-745d-4076-b440-f6f46cd803d6 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:14.354980 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c084f5a-745d-4076-b440-f6f46cd803d6 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:14.356465 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 117/239] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:22:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:22:14.362325 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fa63c7a-7ea0-405d-825c-285deabeb3d2 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:14.362833 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fa63c7a-7ea0-405d-825c-285deabeb3d2 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:14.363325 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fa63c7a-7ea0-405d-825c-285deabeb3d2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:14.389572 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-3fa63c7a-7ea0-405d-825c-285deabeb3d2 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:22:14.391119 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-3fa63c7a-7ea0-405d-825c-285deabeb3d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:22:14.474944 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 123/240] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:22:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 04:22:14.497182 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c084f5a-745d-4076-b440-f6f46cd803d6 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:14.540866 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:14.608584 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fa63c7a-7ea0-405d-825c-285deabeb3d2 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:14.625656 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c084f5a-745d-4076-b440-f6f46cd803d6 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:14.692918 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdf2fee5-1aea-464e-b186-3f09fb06fbac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:14.693563 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdf2fee5-1aea-464e-b186-3f09fb06fbac None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:14.693845 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdf2fee5-1aea-464e-b186-3f09fb06fbac None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:14.694117 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdf2fee5-1aea-464e-b186-3f09fb06fbac None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:14.694259 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:14.715812 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fa63c7a-7ea0-405d-825c-285deabeb3d2 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:14.729573 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdf2fee5-1aea-464e-b186-3f09fb06fbac None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 04:22:14.734516 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-fdf2fee5-1aea-464e-b186-3f09fb06fbac None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 25 04:22:14.734516 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:22:14.734516 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:22:14.734516 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:22:14.734516 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:22:14.734516 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:22:14.734516 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:22:14.734516 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:22:14.734516 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:22:14.734516 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:22:14.734516 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:22:14.734516 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:22:14.734516 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 04:22:14.734516 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 04:22:14.734516 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 04:22:14.734516 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 04:22:14.735410 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 04:22:14.735410 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 04:22:14.735410 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 04:22:14.735410 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 04:22:14.735410 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:22:14.735410 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:22:14.735410 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:22:14.735410 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:22:14.735410 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:22:14.735410 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:22:14.735410 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:22:14.735410 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:22:14.735410 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:22:14.735410 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:22:14.735410 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:22:14.735410 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:22:14.736618 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:22:14.736618 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:22:14.736618 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:22:14.736618 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:22:14.736618 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 04:22:14.736618 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 04:22:14.736618 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 04:22:14.736618 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 04:22:14.736618 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 04:22:14.736618 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 04:22:14.736618 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 25 04:22:14.736618 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:22:14.741861 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 118/241] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Sep 25 04:22:14 2022] GET /identity/v3/roles/reader => generated 84 bytes in 245 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:22:14.746647 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdf2fee5-1aea-464e-b186-3f09fb06fbac None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:14.770530 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01d6bb77-b761-48b9-b200-117be42ae347 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:14.771731 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01d6bb77-b761-48b9-b200-117be42ae347 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:14.772210 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01d6bb77-b761-48b9-b200-117be42ae347 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:14.772770 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01d6bb77-b761-48b9-b200-117be42ae347 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:14.774150 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01d6bb77-b761-48b9-b200-117be42ae347 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:14.775850 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01d6bb77-b761-48b9-b200-117be42ae347 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:14.777849 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 124/242] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:22:14 2022] GET /identity/v3/domains/default => generated 199 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:14.786702 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01d6bb77-b761-48b9-b200-117be42ae347 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:14.916685 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:14.927820 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdf2fee5-1aea-464e-b186-3f09fb06fbac None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:14.934084 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01d6bb77-b761-48b9-b200-117be42ae347 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:14.964100 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ece7a291-0fe2-4967-bea6-57f328dec60f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:14.964753 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ece7a291-0fe2-4967-bea6-57f328dec60f None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:14.965035 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ece7a291-0fe2-4967-bea6-57f328dec60f None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:14.965306 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ece7a291-0fe2-4967-bea6-57f328dec60f None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:14.965967 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ece7a291-0fe2-4967-bea6-57f328dec60f None admin] RBAC: Adding query filter params (name=reader) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:14.966270 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ece7a291-0fe2-4967-bea6-57f328dec60f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:14.967653 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ece7a291-0fe2-4967-bea6-57f328dec60f None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:14.976951 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 119/243] 10.222.0.25 () {56 vars in 1101 bytes} [Sun Sep 25 04:22:14 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 230 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:22:14.977826 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ece7a291-0fe2-4967-bea6-57f328dec60f None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:14.979875 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e90ec43f-8dcc-41c8-a2dd-7e3d1a2a779a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:14.980944 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e90ec43f-8dcc-41c8-a2dd-7e3d1a2a779a None admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:14.982274 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e90ec43f-8dcc-41c8-a2dd-7e3d1a2a779a None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:14.982759 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e90ec43f-8dcc-41c8-a2dd-7e3d1a2a779a None admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:14.983660 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e90ec43f-8dcc-41c8-a2dd-7e3d1a2a779a None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:14.985437 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e90ec43f-8dcc-41c8-a2dd-7e3d1a2a779a None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:15.009623 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:15.022039 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 125/244] 10.222.0.25 () {60 vars in 1132 bytes} [Sun Sep 25 04:22:14 2022] POST /identity/v3/users => generated 293 bytes in 236 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:22:15.029289 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e90ec43f-8dcc-41c8-a2dd-7e3d1a2a779a None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:15.113097 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ece7a291-0fe2-4967-bea6-57f328dec60f None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:15.152921 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22df986b-4be0-4678-80fa-97ea12832982 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:15.154076 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22df986b-4be0-4678-80fa-97ea12832982 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:15.154587 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22df986b-4be0-4678-80fa-97ea12832982 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:15.155054 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22df986b-4be0-4678-80fa-97ea12832982 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:15.167349 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22df986b-4be0-4678-80fa-97ea12832982 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 04:22:15.173800 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-22df986b-4be0-4678-80fa-97ea12832982 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 25 04:22:15.173800 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:22:15.173800 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:22:15.173800 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:22:15.173800 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:22:15.173800 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:22:15.173800 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:22:15.173800 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:22:15.173800 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:22:15.173800 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:22:15.173800 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:22:15.173800 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:22:15.173800 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 04:22:15.173800 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 04:22:15.173800 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 04:22:15.173800 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 04:22:15.174959 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 04:22:15.174959 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 04:22:15.174959 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 04:22:15.174959 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 04:22:15.174959 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:22:15.174959 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:22:15.174959 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:22:15.174959 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:22:15.174959 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:22:15.174959 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:22:15.174959 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:22:15.174959 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:22:15.174959 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:22:15.174959 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:22:15.174959 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:22:15.174959 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:22:15.175805 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:22:15.175805 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:22:15.175805 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:22:15.175805 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:22:15.175805 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 04:22:15.175805 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 04:22:15.175805 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 04:22:15.175805 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 04:22:15.175805 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 04:22:15.175805 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 04:22:15.175805 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 25 04:22:15.175805 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:22:15.178358 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 120/245] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Sep 25 04:22:14 2022] GET /identity/v3/roles/member => generated 84 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:22:15.184220 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22df986b-4be0-4678-80fa-97ea12832982 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:15.203752 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:15.229837 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e90ec43f-8dcc-41c8-a2dd-7e3d1a2a779a None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:15.307977 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22df986b-4be0-4678-80fa-97ea12832982 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:15.315941 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b85127d-6fb2-47de-9f22-d059016c2401 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:15.316592 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b85127d-6fb2-47de-9f22-d059016c2401 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:15.316920 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b85127d-6fb2-47de-9f22-d059016c2401 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:15.317195 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b85127d-6fb2-47de-9f22-d059016c2401 None admin] PATH_INFO: `/v3/users/16fb4cdf8f184df2a8e183c86c41c3bf` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:15.336974 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b85127d-6fb2-47de-9f22-d059016c2401 None admin] RBAC: Authorizing `identity:get_user(user_id=16fb4cdf8f184df2a8e183c86c41c3bf)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:15.338495 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b85127d-6fb2-47de-9f22-d059016c2401 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:15.340534 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 126/246] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Sep 25 04:22:15 2022] GET /identity/v3/users/16fb4cdf8f184df2a8e183c86c41c3bf => generated 297 bytes in 318 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:15.346336 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b85127d-6fb2-47de-9f22-d059016c2401 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:15.349564 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbdf02eb-f0a0-485d-b672-34e9f61fbc3e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:15.350483 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbdf02eb-f0a0-485d-b672-34e9f61fbc3e None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:15.350769 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbdf02eb-f0a0-485d-b672-34e9f61fbc3e None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:15.351047 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbdf02eb-f0a0-485d-b672-34e9f61fbc3e None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:15.351720 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbdf02eb-f0a0-485d-b672-34e9f61fbc3e None admin] RBAC: Adding query filter params (name=member) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:15.352027 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbdf02eb-f0a0-485d-b672-34e9f61fbc3e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:15.353455 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbdf02eb-f0a0-485d-b672-34e9f61fbc3e None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:15.364373 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 121/247] 10.222.0.25 () {56 vars in 1101 bytes} [Sun Sep 25 04:22:15 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:15.369934 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbdf02eb-f0a0-485d-b672-34e9f61fbc3e None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:15.415908 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:15.461800 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:15.474853 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b85127d-6fb2-47de-9f22-d059016c2401 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:15.502359 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbdf02eb-f0a0-485d-b672-34e9f61fbc3e None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:15.529042 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf37799c-dac7-4cff-a111-c4911928cba7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:15.529667 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf37799c-dac7-4cff-a111-c4911928cba7 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:15.529944 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf37799c-dac7-4cff-a111-c4911928cba7 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:15.530664 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf37799c-dac7-4cff-a111-c4911928cba7 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:15.550624 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a0f730d-4560-4122-98ab-4fec8eb99d49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:15.551277 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a0f730d-4560-4122-98ab-4fec8eb99d49 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:15.551562 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a0f730d-4560-4122-98ab-4fec8eb99d49 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:15.551838 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a0f730d-4560-4122-98ab-4fec8eb99d49 None admin] PATH_INFO: `/v3/users/4f5a0c33729d4e7d9813103a3a5acef7` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:15.553035 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a0f730d-4560-4122-98ab-4fec8eb99d49 None admin] RBAC: Authorizing `identity:get_user(user_id=4f5a0c33729d4e7d9813103a3a5acef7)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:15.554737 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a0f730d-4560-4122-98ab-4fec8eb99d49 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:15.556371 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 122/248] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Sep 25 04:22:15 2022] GET /identity/v3/users/4f5a0c33729d4e7d9813103a3a5acef7 => generated 297 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:22:15.557662 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b98047-48a6-4ca2-adf7-c364038b8822 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:15.557947 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b98047-48a6-4ca2-adf7-c364038b8822 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:15.558276 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b98047-48a6-4ca2-adf7-c364038b8822 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:15.559437 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 123/249] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:22:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:22:15.564794 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e81592b2-74cf-4114-b854-427b97d63c37 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:15.565083 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e81592b2-74cf-4114-b854-427b97d63c37 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:15.565357 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e81592b2-74cf-4114-b854-427b97d63c37 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:15.587243 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-e81592b2-74cf-4114-b854-427b97d63c37 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:22:15.588460 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-e81592b2-74cf-4114-b854-427b97d63c37 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:22:15.628501 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf37799c-dac7-4cff-a111-c4911928cba7 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:15.630641 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf37799c-dac7-4cff-a111-c4911928cba7 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:15.683871 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 124/250] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:22:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 25 04:22:15.685924 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e81592b2-74cf-4114-b854-427b97d63c37 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:15.707959 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:15.758772 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 127/251] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Sep 25 04:22:15 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 413 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 25 04:22:15.765833 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf37799c-dac7-4cff-a111-c4911928cba7 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:15.772760 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:15.801072 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e81592b2-74cf-4114-b854-427b97d63c37 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:15.840317 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4022b813-7c65-49c1-915a-f332617708f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:15.841246 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4022b813-7c65-49c1-915a-f332617708f8 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:15.841789 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4022b813-7c65-49c1-915a-f332617708f8 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:15.842206 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4022b813-7c65-49c1-915a-f332617708f8 None admin] PATH_INFO: `/v3/projects/12400184cf3844998e5e6d8078b8cda4` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:15.843483 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4022b813-7c65-49c1-915a-f332617708f8 None admin] RBAC: Authorizing `identity:get_project(project_id=12400184cf3844998e5e6d8078b8cda4)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:15.845220 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4022b813-7c65-49c1-915a-f332617708f8 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:15.847097 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 125/252] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Sep 25 04:22:15 2022] GET /identity/v3/projects/12400184cf3844998e5e6d8078b8cda4 => generated 302 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:15.848462 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f4376eb-594f-4d83-a15a-0f81d67570c6 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:15.848961 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f4376eb-594f-4d83-a15a-0f81d67570c6 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:15.849434 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f4376eb-594f-4d83-a15a-0f81d67570c6 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:15.850905 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 126/253] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:22:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:22:15.856148 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f4376eb-594f-4d83-a15a-0f81d67570c6 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:15.907485 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf37799c-dac7-4cff-a111-c4911928cba7 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:15.978101 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f499f224-7e20-4962-8307-7956b1ffd48f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:15.979360 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f499f224-7e20-4962-8307-7956b1ffd48f None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:15.980464 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f499f224-7e20-4962-8307-7956b1ffd48f None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:15.981111 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f499f224-7e20-4962-8307-7956b1ffd48f None admin] PATH_INFO: `/v3/projects/12400184cf3844998e5e6d8078b8cda4` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:15.984329 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f499f224-7e20-4962-8307-7956b1ffd48f None admin] RBAC: Authorizing `identity:get_project(project_id=12400184cf3844998e5e6d8078b8cda4)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:15.987307 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f499f224-7e20-4962-8307-7956b1ffd48f None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:15.991795 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 128/254] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Sep 25 04:22:15 2022] GET /identity/v3/projects/12400184cf3844998e5e6d8078b8cda4 => generated 302 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:15.993387 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e317bb-583d-4a65-ab9f-d3dbed52b927 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:15.994945 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e317bb-583d-4a65-ab9f-d3dbed52b927 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:15.996432 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e317bb-583d-4a65-ab9f-d3dbed52b927 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:16.027655 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-71e317bb-583d-4a65-ab9f-d3dbed52b927 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:22:16.029711 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-71e317bb-583d-4a65-ab9f-d3dbed52b927 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:22:16.049365 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:16.054968 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f4376eb-594f-4d83-a15a-0f81d67570c6 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:16.093448 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31e5b59b-16b9-4a6c-95f5-64d980014534 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:16.094040 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31e5b59b-16b9-4a6c-95f5-64d980014534 None admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:16.094321 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31e5b59b-16b9-4a6c-95f5-64d980014534 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:16.094593 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31e5b59b-16b9-4a6c-95f5-64d980014534 None admin] PATH_INFO: `/v3/projects/12400184cf3844998e5e6d8078b8cda4/users/4f5a0c33729d4e7d9813103a3a5acef7/roles/8f6cc574056c4278bc9778f14dc4b411` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:16.098245 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31e5b59b-16b9-4a6c-95f5-64d980014534 None admin] RBAC: Authorizing `identity:create_grant(project_id=12400184cf3844998e5e6d8078b8cda4, user_id=4f5a0c33729d4e7d9813103a3a5acef7, role_id=8f6cc574056c4278bc9778f14dc4b411)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:16.099707 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31e5b59b-16b9-4a6c-95f5-64d980014534 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:16.111872 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 129/255] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:22:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 04:22:16.113018 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71e317bb-583d-4a65-ab9f-d3dbed52b927 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:16.115974 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 127/256] 10.222.0.25 () {58 vars in 1439 bytes} [Sun Sep 25 04:22:15 2022] PUT /identity/v3/projects/12400184cf3844998e5e6d8078b8cda4/users/4f5a0c33729d4e7d9813103a3a5acef7/roles/8f6cc574056c4278bc9778f14dc4b411 => generated 0 bytes in 260 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:22:16.120952 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31e5b59b-16b9-4a6c-95f5-64d980014534 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:16.129045 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:16.372489 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71e317bb-583d-4a65-ab9f-d3dbed52b927 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:16.372953 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31e5b59b-16b9-4a6c-95f5-64d980014534 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:16.430984 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6083ed94-a625-4f7c-8d47-bb97eba06caf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:16.431864 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6083ed94-a625-4f7c-8d47-bb97eba06caf None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:16.432158 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6083ed94-a625-4f7c-8d47-bb97eba06caf None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:16.432530 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6083ed94-a625-4f7c-8d47-bb97eba06caf None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:16.435013 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb54d400-08bc-455a-86d2-bf6bf6561dcc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:16.435638 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb54d400-08bc-455a-86d2-bf6bf6561dcc None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:16.435917 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb54d400-08bc-455a-86d2-bf6bf6561dcc None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:16.436190 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb54d400-08bc-455a-86d2-bf6bf6561dcc None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:16.436964 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb54d400-08bc-455a-86d2-bf6bf6561dcc None admin] RBAC: Adding query filter params (role.id=0e4fa3b3402d463da90ddfcd17be7c17, scope.project.id=12400184cf3844998e5e6d8078b8cda4, user.id=16fb4cdf8f184df2a8e183c86c41c3bf) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:16.437278 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb54d400-08bc-455a-86d2-bf6bf6561dcc None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:16.438890 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb54d400-08bc-455a-86d2-bf6bf6561dcc None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:16.445548 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 130/257] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Sep 25 04:22:16 2022] GET /identity/v3/role_assignments?user.id=16fb4cdf8f184df2a8e183c86c41c3bf&scope.project.id=12400184cf3844998e5e6d8078b8cda4&role.id=0e4fa3b3402d463da90ddfcd17be7c17 => generated 259 bytes in 333 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:16.446389 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb54d400-08bc-455a-86d2-bf6bf6561dcc None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:16.465663 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6083ed94-a625-4f7c-8d47-bb97eba06caf None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 04:22:16.472259 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-6083ed94-a625-4f7c-8d47-bb97eba06caf None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 04:22:16.472259 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:22:16.472259 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:22:16.472259 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:22:16.472259 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:22:16.472259 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:22:16.472259 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:22:16.472259 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:22:16.472259 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:22:16.472259 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:22:16.472259 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:22:16.472259 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:22:16.472259 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 04:22:16.472259 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 04:22:16.472259 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 04:22:16.472259 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 04:22:16.473268 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 04:22:16.473268 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 04:22:16.473268 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 04:22:16.473268 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 04:22:16.473268 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:22:16.473268 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:22:16.473268 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:22:16.473268 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:22:16.473268 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:22:16.473268 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:22:16.473268 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:22:16.473268 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:22:16.473268 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:22:16.473268 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:22:16.473268 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:22:16.473268 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:22:16.474083 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:22:16.474083 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:22:16.474083 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:22:16.474083 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:22:16.474083 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 04:22:16.474083 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 04:22:16.474083 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 04:22:16.474083 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 04:22:16.474083 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 04:22:16.474083 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 04:22:16.474083 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 04:22:16.474083 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:22:16.474083 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 128/258] 10.222.0.25 () {56 vars in 1096 bytes} [Sun Sep 25 04:22:16 2022] GET /identity/v3/roles/admin => generated 83 bytes in 357 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:22:16.486407 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:16.501849 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6083ed94-a625-4f7c-8d47-bb97eba06caf None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:16.580650 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb54d400-08bc-455a-86d2-bf6bf6561dcc None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:16.627094 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ed11b75-d3c6-4d62-8e62-b6349b236b9f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:16.628003 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ed11b75-d3c6-4d62-8e62-b6349b236b9f None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:16.628430 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ed11b75-d3c6-4d62-8e62-b6349b236b9f None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:16.628879 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ed11b75-d3c6-4d62-8e62-b6349b236b9f None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:16.650732 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ed11b75-d3c6-4d62-8e62-b6349b236b9f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 04:22:16.664770 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-0ed11b75-d3c6-4d62-8e62-b6349b236b9f None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 25 04:22:16.664770 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:22:16.664770 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:22:16.664770 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:22:16.664770 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:22:16.664770 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:22:16.664770 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:22:16.664770 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:22:16.664770 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:22:16.664770 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:22:16.664770 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:22:16.664770 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:22:16.664770 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 04:22:16.664770 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 04:22:16.664770 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 04:22:16.664770 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 04:22:16.665933 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 04:22:16.665933 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 04:22:16.665933 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 04:22:16.665933 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 04:22:16.665933 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:22:16.665933 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:22:16.665933 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:22:16.665933 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:22:16.665933 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:22:16.665933 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:22:16.665933 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:22:16.665933 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:22:16.665933 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:22:16.665933 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:22:16.665933 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:22:16.665933 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:22:16.666949 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:22:16.666949 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:22:16.666949 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:22:16.666949 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:22:16.666949 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 04:22:16.666949 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 04:22:16.666949 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 04:22:16.666949 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 04:22:16.666949 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 04:22:16.666949 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 04:22:16.666949 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 25 04:22:16.666949 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:22:16.667625 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:16.669461 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 131/259] 10.222.0.25 () {56 vars in 1114 bytes} [Sun Sep 25 04:22:16 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 223 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:22:16.677812 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ed11b75-d3c6-4d62-8e62-b6349b236b9f None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:16.759660 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6083ed94-a625-4f7c-8d47-bb97eba06caf None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:16.778697 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:16.804629 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ed11b75-d3c6-4d62-8e62-b6349b236b9f None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:16.840213 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-933474cf-b535-4301-8c4c-802d5907d093 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:16.841192 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-933474cf-b535-4301-8c4c-802d5907d093 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:16.841655 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-933474cf-b535-4301-8c4c-802d5907d093 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:16.842115 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-933474cf-b535-4301-8c4c-802d5907d093 None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:16.842977 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-933474cf-b535-4301-8c4c-802d5907d093 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:16.843463 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-933474cf-b535-4301-8c4c-802d5907d093 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:16.845565 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-933474cf-b535-4301-8c4c-802d5907d093 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:16.846032 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13e18d46-88c4-4381-8e36-b01c23433844 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:16.846664 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13e18d46-88c4-4381-8e36-b01c23433844 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:16.846951 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13e18d46-88c4-4381-8e36-b01c23433844 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:16.847231 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13e18d46-88c4-4381-8e36-b01c23433844 None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:16.847899 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13e18d46-88c4-4381-8e36-b01c23433844 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:16.848210 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13e18d46-88c4-4381-8e36-b01c23433844 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:16.849696 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13e18d46-88c4-4381-8e36-b01c23433844 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:16.855778 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 132/260] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Sep 25 04:22:16 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:22:16.863462 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 129/261] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Sep 25 04:22:16 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 361 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:16.865184 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-933474cf-b535-4301-8c4c-802d5907d093 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:16.886407 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13e18d46-88c4-4381-8e36-b01c23433844 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:16.977979 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-933474cf-b535-4301-8c4c-802d5907d093 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:17.014632 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-723a560c-174c-4fc0-8074-74296fdc3225 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:17.015576 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-723a560c-174c-4fc0-8074-74296fdc3225 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:17.016039 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-723a560c-174c-4fc0-8074-74296fdc3225 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:17.016492 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-723a560c-174c-4fc0-8074-74296fdc3225 None admin] PATH_INFO: `/v3/users/dbce0115f8df41a2964a6bf6677e25da` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:17.021724 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-723a560c-174c-4fc0-8074-74296fdc3225 None admin] RBAC: Authorizing `identity:get_user(user_id=dbce0115f8df41a2964a6bf6677e25da)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:17.023423 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-723a560c-174c-4fc0-8074-74296fdc3225 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:17.026043 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 133/262] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Sep 25 04:22:16 2022] GET /identity/v3/users/dbce0115f8df41a2964a6bf6677e25da => generated 283 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:17.034449 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-723a560c-174c-4fc0-8074-74296fdc3225 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:17.061982 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:17.122201 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13e18d46-88c4-4381-8e36-b01c23433844 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:17.164583 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:17.190237 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b156664-5cf2-4a57-b4b5-f65d65e8ffa9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:17.191174 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b156664-5cf2-4a57-b4b5-f65d65e8ffa9 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:17.191465 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b156664-5cf2-4a57-b4b5-f65d65e8ffa9 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:17.192181 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b156664-5cf2-4a57-b4b5-f65d65e8ffa9 None admin] PATH_INFO: `/v3/users/dbce0115f8df41a2964a6bf6677e25da` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:17.193513 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b156664-5cf2-4a57-b4b5-f65d65e8ffa9 None admin] RBAC: Authorizing `identity:get_user(user_id=dbce0115f8df41a2964a6bf6677e25da)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:17.194984 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b156664-5cf2-4a57-b4b5-f65d65e8ffa9 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:17.197220 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 130/263] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Sep 25 04:22:16 2022] GET /identity/v3/users/dbce0115f8df41a2964a6bf6677e25da => generated 283 bytes in 311 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:22:17.203521 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b156664-5cf2-4a57-b4b5-f65d65e8ffa9 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:17.332192 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-723a560c-174c-4fc0-8074-74296fdc3225 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:17.344170 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b156664-5cf2-4a57-b4b5-f65d65e8ffa9 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:17.410615 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c517aec-e89f-4d2d-b2c8-c00635e66d36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:17.411252 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c517aec-e89f-4d2d-b2c8-c00635e66d36 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:17.411540 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c517aec-e89f-4d2d-b2c8-c00635e66d36 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:17.411819 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c517aec-e89f-4d2d-b2c8-c00635e66d36 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:17.437599 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6899234-50e8-48a0-bc04-28fd8b4d8d12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:17.438162 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6899234-50e8-48a0-bc04-28fd8b4d8d12 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:17.438448 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6899234-50e8-48a0-bc04-28fd8b4d8d12 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:17.438835 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6899234-50e8-48a0-bc04-28fd8b4d8d12 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:17.533537 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:17.552277 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c517aec-e89f-4d2d-b2c8-c00635e66d36 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:17.554119 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c517aec-e89f-4d2d-b2c8-c00635e66d36 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:17.610872 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6899234-50e8-48a0-bc04-28fd8b4d8d12 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:17.612561 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6899234-50e8-48a0-bc04-28fd8b4d8d12 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:17.631049 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:17.744094 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 131/264] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Sep 25 04:22:17 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 541 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 25 04:22:17.750095 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c517aec-e89f-4d2d-b2c8-c00635e66d36 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:17.815766 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 134/265] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Sep 25 04:22:17 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 781 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 25 04:22:17.821058 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6899234-50e8-48a0-bc04-28fd8b4d8d12 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:17.912149 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:17.928321 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c517aec-e89f-4d2d-b2c8-c00635e66d36 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:18.006365 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6899234-50e8-48a0-bc04-28fd8b4d8d12 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:18.018686 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcaedf56-73fd-4a15-b59c-536436e98962 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:18.019646 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcaedf56-73fd-4a15-b59c-536436e98962 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:18.020127 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcaedf56-73fd-4a15-b59c-536436e98962 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:18.020593 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcaedf56-73fd-4a15-b59c-536436e98962 None admin] PATH_INFO: `/v3/projects/12400184cf3844998e5e6d8078b8cda4` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:18.021983 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcaedf56-73fd-4a15-b59c-536436e98962 None admin] RBAC: Authorizing `identity:get_project(project_id=12400184cf3844998e5e6d8078b8cda4)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:18.023620 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcaedf56-73fd-4a15-b59c-536436e98962 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:18.025785 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 132/266] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Sep 25 04:22:17 2022] GET /identity/v3/projects/12400184cf3844998e5e6d8078b8cda4 => generated 302 bytes in 276 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:18.028674 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:18.032366 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcaedf56-73fd-4a15-b59c-536436e98962 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:18.064752 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f614cad-d7f6-48dc-bf42-edb0dc9e6940 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:18.065464 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f614cad-d7f6-48dc-bf42-edb0dc9e6940 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:18.065676 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f614cad-d7f6-48dc-bf42-edb0dc9e6940 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:18.065947 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f614cad-d7f6-48dc-bf42-edb0dc9e6940 None admin] PATH_INFO: `/v3/projects/12400184cf3844998e5e6d8078b8cda4` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:18.067095 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f614cad-d7f6-48dc-bf42-edb0dc9e6940 None admin] RBAC: Authorizing `identity:get_project(project_id=12400184cf3844998e5e6d8078b8cda4)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:18.068549 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f614cad-d7f6-48dc-bf42-edb0dc9e6940 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:18.076158 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 135/267] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Sep 25 04:22:17 2022] GET /identity/v3/projects/12400184cf3844998e5e6d8078b8cda4 => generated 302 bytes in 250 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:22:18.076295 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f614cad-d7f6-48dc-bf42-edb0dc9e6940 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:18.229096 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f614cad-d7f6-48dc-bf42-edb0dc9e6940 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:18.283585 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfa8d481-00ab-4b67-841a-f7f7309a5057 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:18.284101 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfa8d481-00ab-4b67-841a-f7f7309a5057 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:18.284378 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfa8d481-00ab-4b67-841a-f7f7309a5057 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:18.284669 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfa8d481-00ab-4b67-841a-f7f7309a5057 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:18.285485 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfa8d481-00ab-4b67-841a-f7f7309a5057 None admin] RBAC: Adding query filter params (role.id=02a8d263519b4ff1b511a8577293126c, scope.project.id=12400184cf3844998e5e6d8078b8cda4, user.id=dbce0115f8df41a2964a6bf6677e25da) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:18.285784 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfa8d481-00ab-4b67-841a-f7f7309a5057 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:18.287229 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfa8d481-00ab-4b67-841a-f7f7309a5057 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:18.296167 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcaedf56-73fd-4a15-b59c-536436e98962 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:18.296805 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 136/268] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Sep 25 04:22:18 2022] GET /identity/v3/role_assignments?user.id=dbce0115f8df41a2964a6bf6677e25da&scope.project.id=12400184cf3844998e5e6d8078b8cda4&role.id=02a8d263519b4ff1b511a8577293126c => generated 608 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:22:18.356720 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd44cd40-c074-4319-b0ee-5e2a04d56ef9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:18.357370 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd44cd40-c074-4319-b0ee-5e2a04d56ef9 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:18.357651 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd44cd40-c074-4319-b0ee-5e2a04d56ef9 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:18.357921 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd44cd40-c074-4319-b0ee-5e2a04d56ef9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:18.358972 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd44cd40-c074-4319-b0ee-5e2a04d56ef9 None admin] RBAC: Adding query filter params (role.id=c51fe7c5ad804040879acfb54269455f, scope.project.id=12400184cf3844998e5e6d8078b8cda4, user.id=dbce0115f8df41a2964a6bf6677e25da) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:18.359298 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd44cd40-c074-4319-b0ee-5e2a04d56ef9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:18.361313 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd44cd40-c074-4319-b0ee-5e2a04d56ef9 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:18.368763 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 133/269] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Sep 25 04:22:18 2022] GET /identity/v3/role_assignments?user.id=dbce0115f8df41a2964a6bf6677e25da&scope.project.id=12400184cf3844998e5e6d8078b8cda4&role.id=c51fe7c5ad804040879acfb54269455f => generated 608 bytes in 337 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:18.580877 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85310b12-58ae-4c7e-bf1f-5d615d2995a1 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:18.581199 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85310b12-58ae-4c7e-bf1f-5d615d2995a1 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:18.581375 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85310b12-58ae-4c7e-bf1f-5d615d2995a1 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:18.582967 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 137/270] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:22:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:22:18.589251 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b47230e-84f1-41df-bcbc-c350b7120abd None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:18.589542 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b47230e-84f1-41df-bcbc-c350b7120abd None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:18.589810 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b47230e-84f1-41df-bcbc-c350b7120abd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:18.613172 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-2b47230e-84f1-41df-bcbc-c350b7120abd None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:22:18.614400 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-2b47230e-84f1-41df-bcbc-c350b7120abd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:22:18.636329 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:18.695277 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 134/271] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:22:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 04:22:18.713911 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c26d59a-0b84-4c28-b00b-e6678687bb29 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:18.714504 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c26d59a-0b84-4c28-b00b-e6678687bb29 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:18.714972 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c26d59a-0b84-4c28-b00b-e6678687bb29 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:18.716791 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 138/272] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:22:18 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:22:18.722506 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8529209-ea0f-4d68-8dd3-24fcbcbc1acc None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:18.723059 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8529209-ea0f-4d68-8dd3-24fcbcbc1acc None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:18.723534 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8529209-ea0f-4d68-8dd3-24fcbcbc1acc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:18.743942 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-a8529209-ea0f-4d68-8dd3-24fcbcbc1acc None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:22:18.745510 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-a8529209-ea0f-4d68-8dd3-24fcbcbc1acc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:22:18.831291 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 135/273] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:22:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 04:22:19.042034 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c26d59a-0b84-4c28-b00b-e6678687bb29 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:19.093267 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:19.156660 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8529209-ea0f-4d68-8dd3-24fcbcbc1acc None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:19.162227 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c26d59a-0b84-4c28-b00b-e6678687bb29 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:19.209434 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8808cde-f7f3-48a5-a261-9d7b641200d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:19.210434 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8808cde-f7f3-48a5-a261-9d7b641200d7 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:19.210905 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8808cde-f7f3-48a5-a261-9d7b641200d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:19.211365 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8808cde-f7f3-48a5-a261-9d7b641200d7 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:19.212759 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8808cde-f7f3-48a5-a261-9d7b641200d7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:19.214400 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8808cde-f7f3-48a5-a261-9d7b641200d7 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:19.216163 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 139/274] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:22:19 2022] GET /identity/v3/domains/default => generated 199 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:19.221949 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8808cde-f7f3-48a5-a261-9d7b641200d7 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:19.254404 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:19.275567 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8529209-ea0f-4d68-8dd3-24fcbcbc1acc None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:19.312373 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6f45837-89c5-4f22-899c-4d674eb4fff7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:19.313052 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6f45837-89c5-4f22-899c-4d674eb4fff7 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:19.313561 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6f45837-89c5-4f22-899c-4d674eb4fff7 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:19.313993 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6f45837-89c5-4f22-899c-4d674eb4fff7 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:19.324776 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6f45837-89c5-4f22-899c-4d674eb4fff7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 04:22:19.328266 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8808cde-f7f3-48a5-a261-9d7b641200d7 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:19.330387 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-d6f45837-89c5-4f22-899c-4d674eb4fff7 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 25 04:22:19.330387 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:22:19.330387 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:22:19.330387 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:22:19.330387 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:22:19.330387 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:22:19.330387 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:22:19.330387 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:22:19.330387 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:22:19.330387 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:22:19.330387 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:22:19.330387 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:22:19.330387 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 04:22:19.330387 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 04:22:19.330387 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 04:22:19.330387 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 04:22:19.331304 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 04:22:19.331304 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 04:22:19.331304 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 04:22:19.331304 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 04:22:19.331304 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:22:19.331304 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:22:19.331304 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:22:19.331304 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:22:19.331304 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:22:19.331304 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:22:19.331304 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:22:19.331304 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:22:19.331304 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:22:19.331304 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:22:19.331304 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:22:19.331304 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:22:19.332119 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:22:19.332119 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:22:19.332119 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:22:19.332119 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:22:19.332119 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 04:22:19.332119 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 04:22:19.332119 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 04:22:19.332119 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 04:22:19.332119 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 04:22:19.332119 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 04:22:19.332119 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 25 04:22:19.332119 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:22:19.332119 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 136/275] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Sep 25 04:22:19 2022] GET /identity/v3/roles/member => generated 84 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:22:19.332832 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9763641-4884-44d6-a3bc-b64d34b04ec8 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:19.333760 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9763641-4884-44d6-a3bc-b64d34b04ec8 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:19.333760 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9763641-4884-44d6-a3bc-b64d34b04ec8 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:19.334603 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 137/276] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:22:19 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:22:19.339402 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9763641-4884-44d6-a3bc-b64d34b04ec8 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:19.347116 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:19.374316 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de211044-5a17-4bdb-91e0-37858994b261 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:19.375019 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de211044-5a17-4bdb-91e0-37858994b261 None admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:19.375305 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de211044-5a17-4bdb-91e0-37858994b261 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:19.375577 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de211044-5a17-4bdb-91e0-37858994b261 None admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:19.376327 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de211044-5a17-4bdb-91e0-37858994b261 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:19.377813 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de211044-5a17-4bdb-91e0-37858994b261 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:19.404088 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 140/277] 10.222.0.25 () {60 vars in 1132 bytes} [Sun Sep 25 04:22:19 2022] POST /identity/v3/users => generated 293 bytes in 182 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 04:22:19.408908 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76cfcd72-5f4b-4743-bc68-b246588cd285 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:19.408908 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76cfcd72-5f4b-4743-bc68-b246588cd285 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:19.408908 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76cfcd72-5f4b-4743-bc68-b246588cd285 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:19.436313 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-76cfcd72-5f4b-4743-bc68-b246588cd285 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:22:19.437657 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-76cfcd72-5f4b-4743-bc68-b246588cd285 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:22:19.488671 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9763641-4884-44d6-a3bc-b64d34b04ec8 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:19.524800 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 141/278] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:22:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 04:22:19.531209 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d69175-368c-4e0a-9f92-50a7f0685008 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:19.531498 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d69175-368c-4e0a-9f92-50a7f0685008 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:19.531774 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d69175-368c-4e0a-9f92-50a7f0685008 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:19.538043 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 142/279] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:22:19 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 25 04:22:19.539412 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41a77711-6190-4773-9170-1cdbf47c29fd None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:19.539701 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41a77711-6190-4773-9170-1cdbf47c29fd None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:19.539973 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41a77711-6190-4773-9170-1cdbf47c29fd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:19.545100 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4afc2a1-55bf-4482-842e-4ed645d61f05 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:19.547056 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4afc2a1-55bf-4482-842e-4ed645d61f05 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:19.547540 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4afc2a1-55bf-4482-842e-4ed645d61f05 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:19.549843 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4afc2a1-55bf-4482-842e-4ed645d61f05 None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:19.550937 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4afc2a1-55bf-4482-842e-4ed645d61f05 None admin] RBAC: Adding query filter params (name=member) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:19.551429 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4afc2a1-55bf-4482-842e-4ed645d61f05 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:19.553076 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4afc2a1-55bf-4482-842e-4ed645d61f05 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:19.561272 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-41a77711-6190-4773-9170-1cdbf47c29fd None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:22:19.562506 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-41a77711-6190-4773-9170-1cdbf47c29fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:22:19.564503 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 138/280] 10.222.0.25 () {56 vars in 1101 bytes} [Sun Sep 25 04:22:19 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:19.569845 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4afc2a1-55bf-4482-842e-4ed645d61f05 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:19.583707 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:19.651933 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 143/281] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:22:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 04:22:19.709107 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4afc2a1-55bf-4482-842e-4ed645d61f05 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:19.746751 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23d29b21-8cb0-41f2-aaaa-f26f16ea97f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:19.747450 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23d29b21-8cb0-41f2-aaaa-f26f16ea97f7 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:19.747739 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23d29b21-8cb0-41f2-aaaa-f26f16ea97f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:19.748017 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23d29b21-8cb0-41f2-aaaa-f26f16ea97f7 None admin] PATH_INFO: `/v3/users/3b93d7dd85bf4030a9d909bcc5613fcb` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:19.764568 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23d29b21-8cb0-41f2-aaaa-f26f16ea97f7 None admin] RBAC: Authorizing `identity:get_user(user_id=3b93d7dd85bf4030a9d909bcc5613fcb)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:19.766421 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23d29b21-8cb0-41f2-aaaa-f26f16ea97f7 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:19.768861 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 139/282] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Sep 25 04:22:19 2022] GET /identity/v3/users/3b93d7dd85bf4030a9d909bcc5613fcb => generated 293 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:22:19.774993 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41a77711-6190-4773-9170-1cdbf47c29fd None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:19.804705 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23d29b21-8cb0-41f2-aaaa-f26f16ea97f7 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:19.806064 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:19.910419 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:19.920317 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41a77711-6190-4773-9170-1cdbf47c29fd None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:19.959573 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-080a379f-05a6-44e6-bae4-30e8c3527e0c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:19.960198 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-080a379f-05a6-44e6-bae4-30e8c3527e0c None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:19.960475 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-080a379f-05a6-44e6-bae4-30e8c3527e0c None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:19.964849 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-080a379f-05a6-44e6-bae4-30e8c3527e0c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:19.964849 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-080a379f-05a6-44e6-bae4-30e8c3527e0c None admin] RBAC: Adding query filter params (role.id=8f6cc574056c4278bc9778f14dc4b411, user.id=3b93d7dd85bf4030a9d909bcc5613fcb, scope.system=all) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:19.964849 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-080a379f-05a6-44e6-bae4-30e8c3527e0c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:19.965144 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23d29b21-8cb0-41f2-aaaa-f26f16ea97f7 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:19.965224 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-080a379f-05a6-44e6-bae4-30e8c3527e0c None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:19.970314 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 144/283] 10.222.0.25 () {56 vars in 1308 bytes} [Sun Sep 25 04:22:19 2022] GET /identity/v3/role_assignments?user.id=3b93d7dd85bf4030a9d909bcc5613fcb&scope.system=all&role.id=8f6cc574056c4278bc9778f14dc4b411 => generated 226 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:19.971462 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-080a379f-05a6-44e6-bae4-30e8c3527e0c None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:20.017674 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6065b28-99a6-43d1-8577-1c0c6772c11f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:20.018654 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6065b28-99a6-43d1-8577-1c0c6772c11f None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:20.019084 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6065b28-99a6-43d1-8577-1c0c6772c11f None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:20.019567 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6065b28-99a6-43d1-8577-1c0c6772c11f None admin] PATH_INFO: `/v3/roles/member` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:20.037088 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6065b28-99a6-43d1-8577-1c0c6772c11f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 04:22:20.044753 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-f6065b28-99a6-43d1-8577-1c0c6772c11f None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 25 04:22:20.044753 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:22:20.044753 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:22:20.044753 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:22:20.044753 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:22:20.044753 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:22:20.044753 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:22:20.044753 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:22:20.044753 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:22:20.044753 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:22:20.044753 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:22:20.044753 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:22:20.044753 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 04:22:20.044753 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 04:22:20.044753 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 04:22:20.044753 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 04:22:20.045934 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 04:22:20.045934 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 04:22:20.045934 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 04:22:20.045934 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 04:22:20.045934 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:22:20.045934 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:22:20.045934 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:22:20.045934 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:22:20.045934 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:22:20.045934 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:22:20.045934 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:22:20.045934 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:22:20.045934 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:22:20.045934 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:22:20.045934 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:22:20.045934 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:22:20.046909 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:22:20.046909 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:22:20.046909 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:22:20.046909 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:22:20.046909 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 04:22:20.046909 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 04:22:20.046909 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 04:22:20.046909 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 04:22:20.046909 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 04:22:20.046909 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 04:22:20.046909 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 25 04:22:20.046909 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:22:20.050132 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 140/284] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Sep 25 04:22:19 2022] GET /identity/v3/roles/member => generated 84 bytes in 246 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:22:20.061865 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6065b28-99a6-43d1-8577-1c0c6772c11f None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:20.085407 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:20.110110 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-080a379f-05a6-44e6-bae4-30e8c3527e0c None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:20.187821 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f84b847b-0390-4cfd-bce1-8ff05872f704 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:20.188418 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f84b847b-0390-4cfd-bce1-8ff05872f704 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:20.188784 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f84b847b-0390-4cfd-bce1-8ff05872f704 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:20.189061 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f84b847b-0390-4cfd-bce1-8ff05872f704 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:20.189636 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6065b28-99a6-43d1-8577-1c0c6772c11f None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:20.207286 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f84b847b-0390-4cfd-bce1-8ff05872f704 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 04:22:20.213991 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-f84b847b-0390-4cfd-bce1-8ff05872f704 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 25 04:22:20.213991 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:22:20.213991 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:22:20.213991 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:22:20.213991 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:22:20.213991 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:22:20.213991 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:22:20.213991 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:22:20.213991 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:22:20.213991 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:22:20.213991 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:22:20.213991 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:22:20.213991 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 04:22:20.213991 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 04:22:20.213991 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 04:22:20.213991 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 04:22:20.214939 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 04:22:20.214939 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 04:22:20.214939 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 04:22:20.214939 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 04:22:20.214939 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:22:20.214939 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:22:20.214939 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:22:20.214939 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:22:20.214939 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:22:20.214939 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:22:20.214939 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:22:20.214939 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:22:20.214939 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:22:20.214939 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:22:20.214939 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:22:20.214939 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:22:20.215766 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:22:20.215766 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:22:20.215766 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:22:20.215766 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:22:20.215766 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 04:22:20.215766 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 04:22:20.215766 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 04:22:20.215766 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 04:22:20.215766 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 04:22:20.215766 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 04:22:20.215766 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 25 04:22:20.215766 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:22:20.223248 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 145/285] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Sep 25 04:22:19 2022] GET /identity/v3/roles/reader => generated 84 bytes in 247 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:22:20.224726 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f84b847b-0390-4cfd-bce1-8ff05872f704 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:20.227750 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:20.249436 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f2e2c8f-68e1-4b7c-ab78-ed80ea2bc91f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:20.250057 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f2e2c8f-68e1-4b7c-ab78-ed80ea2bc91f None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:20.250339 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f2e2c8f-68e1-4b7c-ab78-ed80ea2bc91f None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:20.250643 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f2e2c8f-68e1-4b7c-ab78-ed80ea2bc91f None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:20.251321 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f2e2c8f-68e1-4b7c-ab78-ed80ea2bc91f None admin] RBAC: Adding query filter params (name=member) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:20.251626 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f2e2c8f-68e1-4b7c-ab78-ed80ea2bc91f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:20.253056 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f2e2c8f-68e1-4b7c-ab78-ed80ea2bc91f None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:20.271572 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 141/286] 10.222.0.25 () {56 vars in 1101 bytes} [Sun Sep 25 04:22:20 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:20.274811 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f2e2c8f-68e1-4b7c-ab78-ed80ea2bc91f None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:20.410600 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f2e2c8f-68e1-4b7c-ab78-ed80ea2bc91f None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:20.428391 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f84b847b-0390-4cfd-bce1-8ff05872f704 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:20.461936 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5c9184c-15fd-4c5c-8701-4dfbbd1e9b8b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:20.462465 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5c9184c-15fd-4c5c-8701-4dfbbd1e9b8b None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:20.462711 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5c9184c-15fd-4c5c-8701-4dfbbd1e9b8b None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:20.462989 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5c9184c-15fd-4c5c-8701-4dfbbd1e9b8b None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:20.463653 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5c9184c-15fd-4c5c-8701-4dfbbd1e9b8b None admin] RBAC: Adding query filter params (name=reader) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:20.463954 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5c9184c-15fd-4c5c-8701-4dfbbd1e9b8b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:20.468655 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5c9184c-15fd-4c5c-8701-4dfbbd1e9b8b None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:20.475345 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 146/287] 10.222.0.25 () {56 vars in 1101 bytes} [Sun Sep 25 04:22:20 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 251 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:20.481396 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5c9184c-15fd-4c5c-8701-4dfbbd1e9b8b None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:20.490728 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e37b6c5b-c29d-4a75-8a27-81911f7ee5c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:20.491381 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e37b6c5b-c29d-4a75-8a27-81911f7ee5c3 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:20.491662 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e37b6c5b-c29d-4a75-8a27-81911f7ee5c3 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:20.491936 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e37b6c5b-c29d-4a75-8a27-81911f7ee5c3 None admin] PATH_INFO: `/v3/users/4f5a0c33729d4e7d9813103a3a5acef7` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:20.495836 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e37b6c5b-c29d-4a75-8a27-81911f7ee5c3 None admin] RBAC: Authorizing `identity:get_user(user_id=4f5a0c33729d4e7d9813103a3a5acef7)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:20.497352 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e37b6c5b-c29d-4a75-8a27-81911f7ee5c3 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:20.498309 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:20.499952 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 142/288] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Sep 25 04:22:20 2022] GET /identity/v3/users/4f5a0c33729d4e7d9813103a3a5acef7 => generated 297 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:22:20.504368 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e37b6c5b-c29d-4a75-8a27-81911f7ee5c3 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:20.593838 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5c9184c-15fd-4c5c-8701-4dfbbd1e9b8b None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:20.630742 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-577bf9aa-94cd-4266-a9ee-d102cad1f68f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:20.631449 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-577bf9aa-94cd-4266-a9ee-d102cad1f68f None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:20.631738 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-577bf9aa-94cd-4266-a9ee-d102cad1f68f None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:20.632017 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-577bf9aa-94cd-4266-a9ee-d102cad1f68f None admin] PATH_INFO: `/v3/users/16fb4cdf8f184df2a8e183c86c41c3bf` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:20.633215 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-577bf9aa-94cd-4266-a9ee-d102cad1f68f None admin] RBAC: Authorizing `identity:get_user(user_id=16fb4cdf8f184df2a8e183c86c41c3bf)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:20.634563 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:20.634677 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-577bf9aa-94cd-4266-a9ee-d102cad1f68f None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:20.641047 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 147/289] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Sep 25 04:22:20 2022] GET /identity/v3/users/16fb4cdf8f184df2a8e183c86c41c3bf => generated 297 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:20.643550 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-577bf9aa-94cd-4266-a9ee-d102cad1f68f None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:20.745780 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e37b6c5b-c29d-4a75-8a27-81911f7ee5c3 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:20.747791 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-577bf9aa-94cd-4266-a9ee-d102cad1f68f None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:20.783484 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a32423cb-2864-456e-a043-ed2b67412861 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:20.784280 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a32423cb-2864-456e-a043-ed2b67412861 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:20.784567 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a32423cb-2864-456e-a043-ed2b67412861 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:20.784897 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a32423cb-2864-456e-a043-ed2b67412861 None admin] PATH_INFO: `/v3/projects/12400184cf3844998e5e6d8078b8cda4` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:20.786384 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a32423cb-2864-456e-a043-ed2b67412861 None admin] RBAC: Authorizing `identity:get_project(project_id=12400184cf3844998e5e6d8078b8cda4)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:20.786912 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:20.793603 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a32423cb-2864-456e-a043-ed2b67412861 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:20.796436 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 148/290] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Sep 25 04:22:20 2022] GET /identity/v3/projects/12400184cf3844998e5e6d8078b8cda4 => generated 302 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:22:20.802851 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a32423cb-2864-456e-a043-ed2b67412861 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:20.865895 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-070817b7-f536-406b-9316-c6e3b91ed053 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:20.866525 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-070817b7-f536-406b-9316-c6e3b91ed053 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:20.866803 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-070817b7-f536-406b-9316-c6e3b91ed053 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:20.867069 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-070817b7-f536-406b-9316-c6e3b91ed053 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:20.909592 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a32423cb-2864-456e-a043-ed2b67412861 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:20.945809 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25829038-67a3-4b4c-beb7-ff60877292c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:20.946534 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25829038-67a3-4b4c-beb7-ff60877292c8 None admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:20.946819 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25829038-67a3-4b4c-beb7-ff60877292c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:20.947091 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25829038-67a3-4b4c-beb7-ff60877292c8 None admin] PATH_INFO: `/v3/projects/12400184cf3844998e5e6d8078b8cda4/users/16fb4cdf8f184df2a8e183c86c41c3bf/roles/0e4fa3b3402d463da90ddfcd17be7c17` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:20.949122 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25829038-67a3-4b4c-beb7-ff60877292c8 None admin] RBAC: Authorizing `identity:create_grant(project_id=12400184cf3844998e5e6d8078b8cda4, user_id=16fb4cdf8f184df2a8e183c86c41c3bf, role_id=0e4fa3b3402d463da90ddfcd17be7c17)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:20.950831 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25829038-67a3-4b4c-beb7-ff60877292c8 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:20.966017 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 149/291] 10.222.0.25 () {58 vars in 1439 bytes} [Sun Sep 25 04:22:20 2022] PUT /identity/v3/projects/12400184cf3844998e5e6d8078b8cda4/users/16fb4cdf8f184df2a8e183c86c41c3bf/roles/0e4fa3b3402d463da90ddfcd17be7c17 => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 04:22:21.038052 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-070817b7-f536-406b-9316-c6e3b91ed053 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:21.039676 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-070817b7-f536-406b-9316-c6e3b91ed053 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:21.131432 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:21.199326 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 143/292] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Sep 25 04:22:20 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 695 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 25 04:22:21.205762 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25829038-67a3-4b4c-beb7-ff60877292c8 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:21.216238 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:21.319165 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25829038-67a3-4b4c-beb7-ff60877292c8 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:21.367634 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe0b67e9-8acf-4950-a3f4-e4843431d8d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:21.368267 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe0b67e9-8acf-4950-a3f4-e4843431d8d6 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:21.368546 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe0b67e9-8acf-4950-a3f4-e4843431d8d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:21.368836 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe0b67e9-8acf-4950-a3f4-e4843431d8d6 None admin] PATH_INFO: `/v3/projects/12400184cf3844998e5e6d8078b8cda4` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:21.370024 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe0b67e9-8acf-4950-a3f4-e4843431d8d6 None admin] RBAC: Authorizing `identity:get_project(project_id=12400184cf3844998e5e6d8078b8cda4)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:21.371511 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe0b67e9-8acf-4950-a3f4-e4843431d8d6 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:21.373151 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 150/293] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Sep 25 04:22:21 2022] GET /identity/v3/projects/12400184cf3844998e5e6d8078b8cda4 => generated 302 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:21.379986 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-070817b7-f536-406b-9316-c6e3b91ed053 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:21.544472 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-070817b7-f536-406b-9316-c6e3b91ed053 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:21.593611 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46c5a91d-ec09-43b9-8f80-46b4539c4485 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:21.594150 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c5a91d-ec09-43b9-8f80-46b4539c4485 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:21.594431 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c5a91d-ec09-43b9-8f80-46b4539c4485 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:21.594764 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c5a91d-ec09-43b9-8f80-46b4539c4485 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:21.595601 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46c5a91d-ec09-43b9-8f80-46b4539c4485 None admin] RBAC: Adding query filter params (role.id=8f6cc574056c4278bc9778f14dc4b411, scope.project.id=12400184cf3844998e5e6d8078b8cda4, user.id=4f5a0c33729d4e7d9813103a3a5acef7) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:21.595920 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46c5a91d-ec09-43b9-8f80-46b4539c4485 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:21.597401 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46c5a91d-ec09-43b9-8f80-46b4539c4485 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:21.606701 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 144/294] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Sep 25 04:22:21 2022] GET /identity/v3/role_assignments?user.id=4f5a0c33729d4e7d9813103a3a5acef7&scope.project.id=12400184cf3844998e5e6d8078b8cda4&role.id=8f6cc574056c4278bc9778f14dc4b411 => generated 608 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:22.257197 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-683c83d0-583a-4e61-af6f-3c1ac33d6a92 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:22.257743 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-683c83d0-583a-4e61-af6f-3c1ac33d6a92 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:22.257743 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-683c83d0-583a-4e61-af6f-3c1ac33d6a92 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:22.259317 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 151/295] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:22:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:22:22.264487 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bba1d0f-e27c-464b-a3d6-d73ca2c444e0 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:22.264802 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bba1d0f-e27c-464b-a3d6-d73ca2c444e0 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:22.265082 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bba1d0f-e27c-464b-a3d6-d73ca2c444e0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:22.288905 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-973e69cf-572d-4e44-8b7c-54c1237edee9 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:22.289207 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-6bba1d0f-e27c-464b-a3d6-d73ca2c444e0 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:22:22.289675 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-973e69cf-572d-4e44-8b7c-54c1237edee9 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:22.290136 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-973e69cf-572d-4e44-8b7c-54c1237edee9 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:22.290339 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-6bba1d0f-e27c-464b-a3d6-d73ca2c444e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:22:22.292093 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 152/296] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:22:22 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:22:22.297547 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f40675f-742d-48a7-a2a7-2ff12649ec17 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:22.298233 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f40675f-742d-48a7-a2a7-2ff12649ec17 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:22.298709 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f40675f-742d-48a7-a2a7-2ff12649ec17 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:22.310277 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:22.320997 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-0f40675f-742d-48a7-a2a7-2ff12649ec17 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:22:22.322637 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-0f40675f-742d-48a7-a2a7-2ff12649ec17 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:22:22.373053 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 145/297] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:22:22 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 04:22:22.405049 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 153/298] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:22:22 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 25 04:22:22.642212 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bba1d0f-e27c-464b-a3d6-d73ca2c444e0 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:22.658396 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eed69cd-1572-445c-b38c-2f84ba5048f8 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:22.658956 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eed69cd-1572-445c-b38c-2f84ba5048f8 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:22.659451 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eed69cd-1572-445c-b38c-2f84ba5048f8 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:22.661252 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 154/299] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:22:22 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 25 04:22:22.666166 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a2ce9cd-f581-486d-8240-b400c52b0455 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:22.666680 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a2ce9cd-f581-486d-8240-b400c52b0455 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:22.667146 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a2ce9cd-f581-486d-8240-b400c52b0455 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:22.675497 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:22.686394 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-6a2ce9cd-f581-486d-8240-b400c52b0455 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:22:22.687834 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-6a2ce9cd-f581-486d-8240-b400c52b0455 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:22:22.746130 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bba1d0f-e27c-464b-a3d6-d73ca2c444e0 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:22.761370 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 155/300] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:22:22 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 04:22:22.762509 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a2ce9cd-f581-486d-8240-b400c52b0455 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:22.796911 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bedd3c51-5248-41f3-94e4-d5f0ef69716e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:22.798192 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bedd3c51-5248-41f3-94e4-d5f0ef69716e None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:22.799621 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bedd3c51-5248-41f3-94e4-d5f0ef69716e None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:22.800106 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bedd3c51-5248-41f3-94e4-d5f0ef69716e None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:22.813270 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bedd3c51-5248-41f3-94e4-d5f0ef69716e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 04:22:22.820525 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-bedd3c51-5248-41f3-94e4-d5f0ef69716e None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 25 04:22:22.820525 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:22:22.820525 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:22:22.820525 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:22:22.820525 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:22:22.820525 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:22:22.820525 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:22:22.820525 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:22:22.820525 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:22:22.820525 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:22:22.820525 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:22:22.820525 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:22:22.820525 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 04:22:22.820525 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 04:22:22.820525 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 04:22:22.820525 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 04:22:22.821468 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 04:22:22.821468 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 04:22:22.821468 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 04:22:22.821468 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 04:22:22.821468 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:22:22.821468 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:22:22.821468 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:22:22.821468 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:22:22.821468 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:22:22.821468 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:22:22.821468 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:22:22.821468 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:22:22.821468 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:22:22.821468 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:22:22.821468 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:22:22.821468 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:22:22.822291 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:22:22.822291 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:22:22.822291 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:22:22.822291 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:22:22.822291 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 04:22:22.822291 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 04:22:22.822291 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 04:22:22.822291 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 04:22:22.822291 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 04:22:22.822291 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 04:22:22.822291 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 25 04:22:22.822291 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:22:22.824860 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 146/301] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Sep 25 04:22:22 2022] GET /identity/v3/roles/reader => generated 84 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:22:22.831458 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bedd3c51-5248-41f3-94e4-d5f0ef69716e None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:22.842746 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:22.880304 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a2ce9cd-f581-486d-8240-b400c52b0455 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:22.918407 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b80fded4-dac5-4f80-ad3d-23fd44f537bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:22.919386 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b80fded4-dac5-4f80-ad3d-23fd44f537bb None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:22.919877 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b80fded4-dac5-4f80-ad3d-23fd44f537bb None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:22.920343 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b80fded4-dac5-4f80-ad3d-23fd44f537bb None admin] PATH_INFO: `/v3/domains/default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:22.921690 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b80fded4-dac5-4f80-ad3d-23fd44f537bb None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:22.923376 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b80fded4-dac5-4f80-ad3d-23fd44f537bb None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:22.925249 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 156/302] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:22:22 2022] GET /identity/v3/domains/default => generated 199 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:22.932248 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b80fded4-dac5-4f80-ad3d-23fd44f537bb None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:22.936145 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:22.953183 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bedd3c51-5248-41f3-94e4-d5f0ef69716e None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:22.989574 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd9501f3-1725-43d6-b503-51ae0fd432e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:22.990467 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd9501f3-1725-43d6-b503-51ae0fd432e8 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:22.990956 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd9501f3-1725-43d6-b503-51ae0fd432e8 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:22.991425 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd9501f3-1725-43d6-b503-51ae0fd432e8 None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:22.992272 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd9501f3-1725-43d6-b503-51ae0fd432e8 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:22.992821 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd9501f3-1725-43d6-b503-51ae0fd432e8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:22.994385 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd9501f3-1725-43d6-b503-51ae0fd432e8 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:23.004021 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 147/303] 10.222.0.25 () {56 vars in 1101 bytes} [Sun Sep 25 04:22:22 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:22:23.009563 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd9501f3-1725-43d6-b503-51ae0fd432e8 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:23.057529 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b80fded4-dac5-4f80-ad3d-23fd44f537bb None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:23.085923 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:23.097752 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d2c65db-5241-4240-9d43-366d4516de11 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:23.098371 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d2c65db-5241-4240-9d43-366d4516de11 None admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:23.098648 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d2c65db-5241-4240-9d43-366d4516de11 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:23.098976 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d2c65db-5241-4240-9d43-366d4516de11 None admin] PATH_INFO: `/v3/groups` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:23.099608 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d2c65db-5241-4240-9d43-366d4516de11 None admin] RBAC: Authorizing `identity:create_group()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:23.101154 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d2c65db-5241-4240-9d43-366d4516de11 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:23.128673 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 157/304] 10.222.0.25 () {60 vars in 1134 bytes} [Sun Sep 25 04:22:22 2022] POST /identity/v3/groups => generated 229 bytes in 187 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:22:23.128673 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d2c65db-5241-4240-9d43-366d4516de11 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:23.134656 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd9501f3-1725-43d6-b503-51ae0fd432e8 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:23.173495 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ad71018-fb42-4d6a-a5a3-3ac889c281e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:23.174061 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ad71018-fb42-4d6a-a5a3-3ac889c281e7 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:23.174361 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ad71018-fb42-4d6a-a5a3-3ac889c281e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:23.174711 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ad71018-fb42-4d6a-a5a3-3ac889c281e7 None admin] PATH_INFO: `/v3/users/4d34992c3a9c4944a526d17d2b16acfb` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:23.191344 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ad71018-fb42-4d6a-a5a3-3ac889c281e7 None admin] RBAC: Authorizing `identity:get_user(user_id=4d34992c3a9c4944a526d17d2b16acfb)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:23.193174 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ad71018-fb42-4d6a-a5a3-3ac889c281e7 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:23.195367 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 148/305] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Sep 25 04:22:23 2022] GET /identity/v3/users/4d34992c3a9c4944a526d17d2b16acfb => generated 293 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:23.201761 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ad71018-fb42-4d6a-a5a3-3ac889c281e7 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:23.205872 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:23.263056 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d2c65db-5241-4240-9d43-366d4516de11 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:23.311114 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66aee4c8-ec0a-412b-af51-089a01977f0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:23.312311 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66aee4c8-ec0a-412b-af51-089a01977f0d None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:23.312771 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66aee4c8-ec0a-412b-af51-089a01977f0d None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:23.313213 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66aee4c8-ec0a-412b-af51-089a01977f0d None admin] PATH_INFO: `/v3/roles/member` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:23.324907 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ad71018-fb42-4d6a-a5a3-3ac889c281e7 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:23.329508 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66aee4c8-ec0a-412b-af51-089a01977f0d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 04:22:23.334884 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-66aee4c8-ec0a-412b-af51-089a01977f0d None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 25 04:22:23.334884 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:22:23.334884 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:22:23.334884 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:22:23.334884 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:22:23.334884 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:22:23.334884 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:22:23.334884 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:22:23.334884 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:22:23.334884 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:22:23.334884 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:22:23.334884 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:22:23.334884 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 04:22:23.334884 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 04:22:23.334884 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 04:22:23.334884 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 04:22:23.335785 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 04:22:23.335785 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 04:22:23.335785 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 04:22:23.335785 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 04:22:23.335785 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:22:23.335785 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:22:23.335785 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:22:23.335785 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:22:23.335785 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:22:23.335785 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:22:23.335785 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:22:23.335785 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:22:23.335785 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:22:23.335785 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:22:23.335785 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:22:23.335785 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:22:23.336596 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:22:23.336596 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:22:23.336596 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:22:23.336596 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:22:23.336596 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 04:22:23.336596 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 04:22:23.336596 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 04:22:23.336596 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 04:22:23.336596 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 04:22:23.336596 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 04:22:23.336596 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 25 04:22:23.336596 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:22:23.337288 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 158/306] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Sep 25 04:22:23 2022] GET /identity/v3/roles/member => generated 84 bytes in 217 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:22:23.344721 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66aee4c8-ec0a-412b-af51-089a01977f0d None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:23.357982 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:23.369362 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfffda4b-b44e-4670-8c76-23cd28adf094 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:23.369972 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfffda4b-b44e-4670-8c76-23cd28adf094 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:23.370871 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfffda4b-b44e-4670-8c76-23cd28adf094 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:23.370871 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfffda4b-b44e-4670-8c76-23cd28adf094 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:23.371235 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfffda4b-b44e-4670-8c76-23cd28adf094 None admin] RBAC: Adding query filter params (role.id=0e4fa3b3402d463da90ddfcd17be7c17, user.id=4d34992c3a9c4944a526d17d2b16acfb, scope.system=all) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:23.371543 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfffda4b-b44e-4670-8c76-23cd28adf094 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:23.373040 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfffda4b-b44e-4670-8c76-23cd28adf094 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:23.378950 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 149/307] 10.222.0.25 () {56 vars in 1308 bytes} [Sun Sep 25 04:22:23 2022] GET /identity/v3/role_assignments?user.id=4d34992c3a9c4944a526d17d2b16acfb&scope.system=all&role.id=0e4fa3b3402d463da90ddfcd17be7c17 => generated 226 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:23.483578 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66aee4c8-ec0a-412b-af51-089a01977f0d None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:23.513441 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:23.523721 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-885ade2f-93e7-45e4-a932-b92e14ab470d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:23.524339 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-885ade2f-93e7-45e4-a932-b92e14ab470d None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:23.524614 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-885ade2f-93e7-45e4-a932-b92e14ab470d None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:23.524910 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-885ade2f-93e7-45e4-a932-b92e14ab470d None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:23.525576 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-885ade2f-93e7-45e4-a932-b92e14ab470d None admin] RBAC: Adding query filter params (name=member) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:23.525876 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-885ade2f-93e7-45e4-a932-b92e14ab470d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:23.527268 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-885ade2f-93e7-45e4-a932-b92e14ab470d None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:23.538498 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 159/308] 10.222.0.25 () {56 vars in 1101 bytes} [Sun Sep 25 04:22:23 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:23.543966 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfffda4b-b44e-4670-8c76-23cd28adf094 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:23.549666 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ee58cd-bfab-4705-9f31-c63d49cd0cb8 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:23.549962 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ee58cd-bfab-4705-9f31-c63d49cd0cb8 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:23.550222 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ee58cd-bfab-4705-9f31-c63d49cd0cb8 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:23.551863 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 160/309] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:22:23 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 25 04:22:23.557380 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-205a4345-fc9e-41b9-b867-b1ccd2a4e057 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:23.557670 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-205a4345-fc9e-41b9-b867-b1ccd2a4e057 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:23.557950 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-205a4345-fc9e-41b9-b867-b1ccd2a4e057 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:23.578488 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-205a4345-fc9e-41b9-b867-b1ccd2a4e057 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:22:23.579803 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-205a4345-fc9e-41b9-b867-b1ccd2a4e057 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:22:23.655107 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 161/310] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:22:23 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 25 04:22:23.667341 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfffda4b-b44e-4670-8c76-23cd28adf094 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:23.709873 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d19e006b-3626-4c9a-910c-a1c92e61df9f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:23.710854 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d19e006b-3626-4c9a-910c-a1c92e61df9f None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:23.711355 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d19e006b-3626-4c9a-910c-a1c92e61df9f None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:23.711823 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d19e006b-3626-4c9a-910c-a1c92e61df9f None admin] PATH_INFO: `/v3/users/3b93d7dd85bf4030a9d909bcc5613fcb` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:23.713245 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d19e006b-3626-4c9a-910c-a1c92e61df9f None admin] RBAC: Authorizing `identity:get_user(user_id=3b93d7dd85bf4030a9d909bcc5613fcb)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:23.714903 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d19e006b-3626-4c9a-910c-a1c92e61df9f None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:23.716769 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 150/311] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Sep 25 04:22:23 2022] GET /identity/v3/users/3b93d7dd85bf4030a9d909bcc5613fcb => generated 293 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:23.725681 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-205a4345-fc9e-41b9-b867-b1ccd2a4e057 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:23.835265 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-205a4345-fc9e-41b9-b867-b1ccd2a4e057 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:23.850129 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:23.872284 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6d3b2ae-d66c-4fb1-ae67-c3709247b592 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:23.873325 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6d3b2ae-d66c-4fb1-ae67-c3709247b592 None admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:23.873823 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6d3b2ae-d66c-4fb1-ae67-c3709247b592 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:23.874292 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6d3b2ae-d66c-4fb1-ae67-c3709247b592 None admin] PATH_INFO: `/v3/system/users/3b93d7dd85bf4030a9d909bcc5613fcb/roles/8f6cc574056c4278bc9778f14dc4b411` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:23.876035 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6d3b2ae-d66c-4fb1-ae67-c3709247b592 None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=3b93d7dd85bf4030a9d909bcc5613fcb, role_id=8f6cc574056c4278bc9778f14dc4b411)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:23.877678 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6d3b2ae-d66c-4fb1-ae67-c3709247b592 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:23.888282 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 162/312] 10.222.0.25 () {58 vars in 1334 bytes} [Sun Sep 25 04:22:23 2022] PUT /identity/v3/system/users/3b93d7dd85bf4030a9d909bcc5613fcb/roles/8f6cc574056c4278bc9778f14dc4b411 => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:22:23.928012 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d19e006b-3626-4c9a-910c-a1c92e61df9f None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:23.999524 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:24.047166 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d19e006b-3626-4c9a-910c-a1c92e61df9f None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:24.085548 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77b67618-2bdc-4c0e-a0a0-ceb7feb19a83 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:24.086409 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b67618-2bdc-4c0e-a0a0-ceb7feb19a83 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:24.086849 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b67618-2bdc-4c0e-a0a0-ceb7feb19a83 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:24.087261 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b67618-2bdc-4c0e-a0a0-ceb7feb19a83 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:24.098239 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77b67618-2bdc-4c0e-a0a0-ceb7feb19a83 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 04:22:24.103310 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-77b67618-2bdc-4c0e-a0a0-ceb7feb19a83 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 25 04:22:24.103310 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:22:24.103310 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:22:24.103310 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:22:24.103310 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:22:24.103310 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:22:24.103310 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:22:24.103310 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:22:24.103310 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:22:24.103310 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:22:24.103310 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:22:24.103310 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:22:24.103310 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 04:22:24.103310 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 04:22:24.103310 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 04:22:24.103310 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 04:22:24.104408 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 04:22:24.104408 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 04:22:24.104408 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 04:22:24.104408 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 04:22:24.104408 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:22:24.104408 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:22:24.104408 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:22:24.104408 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:22:24.104408 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:22:24.104408 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:22:24.104408 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:22:24.104408 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:22:24.104408 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:22:24.104408 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:22:24.104408 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:22:24.104408 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:22:24.105437 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:22:24.105437 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:22:24.105437 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:22:24.105437 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:22:24.105437 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 04:22:24.105437 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 04:22:24.105437 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 04:22:24.105437 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 04:22:24.105437 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 04:22:24.105437 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 04:22:24.105437 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 25 04:22:24.105437 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:22:24.107713 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 151/313] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Sep 25 04:22:23 2022] GET /identity/v3/roles/reader => generated 84 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:22:24.113869 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6d3b2ae-d66c-4fb1-ae67-c3709247b592 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:24.223537 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6d3b2ae-d66c-4fb1-ae67-c3709247b592 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:24.258508 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7445b40-3bd3-4086-a376-f89aa7232e44 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:24.259355 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7445b40-3bd3-4086-a376-f89aa7232e44 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:24.259803 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7445b40-3bd3-4086-a376-f89aa7232e44 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:24.260216 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7445b40-3bd3-4086-a376-f89aa7232e44 None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:24.261072 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7445b40-3bd3-4086-a376-f89aa7232e44 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:24.261517 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7445b40-3bd3-4086-a376-f89aa7232e44 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:24.263025 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7445b40-3bd3-4086-a376-f89aa7232e44 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:24.272043 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 163/314] 10.222.0.25 () {56 vars in 1101 bytes} [Sun Sep 25 04:22:24 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:24.281722 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77b67618-2bdc-4c0e-a0a0-ceb7feb19a83 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:24.385934 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77b67618-2bdc-4c0e-a0a0-ceb7feb19a83 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:24.410386 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:24.430405 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3999103c-7269-4152-baa0-98496269c9cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:24.431046 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3999103c-7269-4152-baa0-98496269c9cd None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:24.431322 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3999103c-7269-4152-baa0-98496269c9cd None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:24.431591 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3999103c-7269-4152-baa0-98496269c9cd None admin] PATH_INFO: `/v3/users/16fb4cdf8f184df2a8e183c86c41c3bf` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:24.432739 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3999103c-7269-4152-baa0-98496269c9cd None admin] RBAC: Authorizing `identity:get_user(user_id=16fb4cdf8f184df2a8e183c86c41c3bf)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:24.434182 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3999103c-7269-4152-baa0-98496269c9cd None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:24.435740 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 152/315] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Sep 25 04:22:24 2022] GET /identity/v3/users/16fb4cdf8f184df2a8e183c86c41c3bf => generated 297 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:24.442217 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7445b40-3bd3-4086-a376-f89aa7232e44 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:24.495807 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:24.560951 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7445b40-3bd3-4086-a376-f89aa7232e44 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:24.597074 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bba02499-67a7-4a32-baa4-1f7cc76a3f1c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:24.597687 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bba02499-67a7-4a32-baa4-1f7cc76a3f1c None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:24.597960 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bba02499-67a7-4a32-baa4-1f7cc76a3f1c None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:24.598227 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bba02499-67a7-4a32-baa4-1f7cc76a3f1c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:24.667656 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bba02499-67a7-4a32-baa4-1f7cc76a3f1c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:24.669495 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bba02499-67a7-4a32-baa4-1f7cc76a3f1c None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:24.716861 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:24.777232 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 164/316] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Sep 25 04:22:24 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 335 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 25 04:22:24.783915 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3999103c-7269-4152-baa0-98496269c9cd None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:24.896150 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3999103c-7269-4152-baa0-98496269c9cd None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:24.932090 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f1e38a2-9c0a-46f4-baf9-5619a3ad6321 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:24.932664 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f1e38a2-9c0a-46f4-baf9-5619a3ad6321 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:24.932951 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f1e38a2-9c0a-46f4-baf9-5619a3ad6321 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:24.933222 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f1e38a2-9c0a-46f4-baf9-5619a3ad6321 None admin] PATH_INFO: `/v3/projects/12400184cf3844998e5e6d8078b8cda4` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:24.934335 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f1e38a2-9c0a-46f4-baf9-5619a3ad6321 None admin] RBAC: Authorizing `identity:get_project(project_id=12400184cf3844998e5e6d8078b8cda4)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:24.935798 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f1e38a2-9c0a-46f4-baf9-5619a3ad6321 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:24.937726 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 153/317] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Sep 25 04:22:24 2022] GET /identity/v3/projects/12400184cf3844998e5e6d8078b8cda4 => generated 302 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:24.943904 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bba02499-67a7-4a32-baa4-1f7cc76a3f1c None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:25.045967 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bba02499-67a7-4a32-baa4-1f7cc76a3f1c None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:25.086054 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8efbc22a-4a9f-4da3-a6ce-e20905567e48 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:25.086731 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8efbc22a-4a9f-4da3-a6ce-e20905567e48 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:25.087015 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8efbc22a-4a9f-4da3-a6ce-e20905567e48 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:25.087336 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8efbc22a-4a9f-4da3-a6ce-e20905567e48 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:25.088095 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8efbc22a-4a9f-4da3-a6ce-e20905567e48 None admin] RBAC: Adding query filter params (role.id=0e4fa3b3402d463da90ddfcd17be7c17, scope.project.id=12400184cf3844998e5e6d8078b8cda4, user.id=16fb4cdf8f184df2a8e183c86c41c3bf) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:25.088404 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8efbc22a-4a9f-4da3-a6ce-e20905567e48 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:25.089868 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8efbc22a-4a9f-4da3-a6ce-e20905567e48 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:25.096493 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 165/318] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Sep 25 04:22:24 2022] GET /identity/v3/role_assignments?user.id=16fb4cdf8f184df2a8e183c86c41c3bf&scope.project.id=12400184cf3844998e5e6d8078b8cda4&role.id=0e4fa3b3402d463da90ddfcd17be7c17 => generated 608 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:25.453384 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46294844-6317-4bde-ac2e-7f425cc59e17 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:25.453674 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46294844-6317-4bde-ac2e-7f425cc59e17 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:25.454489 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46294844-6317-4bde-ac2e-7f425cc59e17 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:25.455931 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 154/319] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:22:25 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:22:25.461250 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b786b14d-b2ea-49d5-aae4-74c4b4b487fe None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:25.461709 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b786b14d-b2ea-49d5-aae4-74c4b4b487fe None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:25.461787 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b786b14d-b2ea-49d5-aae4-74c4b4b487fe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:25.470834 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:25.481413 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-b786b14d-b2ea-49d5-aae4-74c4b4b487fe None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:22:25.483247 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-b786b14d-b2ea-49d5-aae4-74c4b4b487fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:22:25.553459 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 166/320] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:22:25 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 04:22:25.781860 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ad36439-9a2f-4ab9-b0fe-058b2105e02d None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:25.782719 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ad36439-9a2f-4ab9-b0fe-058b2105e02d None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:25.783185 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ad36439-9a2f-4ab9-b0fe-058b2105e02d None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:25.784604 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 155/321] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:22:25 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:22:25.789715 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf538cbc-072f-428c-9ccf-ec0c2e46efb2 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:25.790005 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf538cbc-072f-428c-9ccf-ec0c2e46efb2 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:25.790278 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf538cbc-072f-428c-9ccf-ec0c2e46efb2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:25.808457 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-bf538cbc-072f-428c-9ccf-ec0c2e46efb2 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:22:25.809654 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-bf538cbc-072f-428c-9ccf-ec0c2e46efb2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:22:25.826367 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ad36439-9a2f-4ab9-b0fe-058b2105e02d None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:25.880279 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 167/322] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:22:25 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 04:22:25.887707 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:25.935333 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ad36439-9a2f-4ab9-b0fe-058b2105e02d None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:25.977967 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-176889b2-8b25-4d2a-a555-5177542e0f8a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:25.978639 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-176889b2-8b25-4d2a-a555-5177542e0f8a None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:25.978921 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-176889b2-8b25-4d2a-a555-5177542e0f8a None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:25.979197 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-176889b2-8b25-4d2a-a555-5177542e0f8a None admin] PATH_INFO: `/v3/domains/default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:25.980413 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-176889b2-8b25-4d2a-a555-5177542e0f8a None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:25.981955 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-176889b2-8b25-4d2a-a555-5177542e0f8a None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:25.983805 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 156/323] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:22:25 2022] GET /identity/v3/domains/default => generated 199 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:22:25.989489 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf538cbc-072f-428c-9ccf-ec0c2e46efb2 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:26.054777 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:26.094466 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf538cbc-072f-428c-9ccf-ec0c2e46efb2 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:26.103145 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b4c976-7491-4e2e-aea9-4ae9ebe75eac None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:26.103429 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b4c976-7491-4e2e-aea9-4ae9ebe75eac None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:26.103702 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b4c976-7491-4e2e-aea9-4ae9ebe75eac None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:26.105229 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 157/324] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:22:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 25 04:22:26.110009 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24ffd0fb-3279-4625-ae77-7a1686ea92ae None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:26.110341 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24ffd0fb-3279-4625-ae77-7a1686ea92ae None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:26.110622 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24ffd0fb-3279-4625-ae77-7a1686ea92ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:26.129400 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09a01c6f-0dca-455d-8680-4d8ad0916cd2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:26.129686 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-24ffd0fb-3279-4625-ae77-7a1686ea92ae None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:22:26.130016 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09a01c6f-0dca-455d-8680-4d8ad0916cd2 None admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:26.130296 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09a01c6f-0dca-455d-8680-4d8ad0916cd2 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:26.130568 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09a01c6f-0dca-455d-8680-4d8ad0916cd2 None admin] PATH_INFO: `/v3/groups` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:26.130810 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-24ffd0fb-3279-4625-ae77-7a1686ea92ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:22:26.131548 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09a01c6f-0dca-455d-8680-4d8ad0916cd2 None admin] RBAC: Authorizing `identity:create_group()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:26.133029 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09a01c6f-0dca-455d-8680-4d8ad0916cd2 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:26.149704 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 168/325] 10.222.0.25 () {60 vars in 1134 bytes} [Sun Sep 25 04:22:25 2022] POST /identity/v3/groups => generated 226 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:22:26.150533 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09a01c6f-0dca-455d-8680-4d8ad0916cd2 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:26.206700 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 158/326] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:22:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 25 04:22:26.263856 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09a01c6f-0dca-455d-8680-4d8ad0916cd2 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:26.289909 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:26.303213 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9731f65-563b-4018-8de7-cdd741306585 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:26.303881 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9731f65-563b-4018-8de7-cdd741306585 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:26.304159 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9731f65-563b-4018-8de7-cdd741306585 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:26.304426 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9731f65-563b-4018-8de7-cdd741306585 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:26.316137 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9731f65-563b-4018-8de7-cdd741306585 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 04:22:26.321608 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-c9731f65-563b-4018-8de7-cdd741306585 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 25 04:22:26.321608 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:22:26.321608 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:22:26.321608 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:22:26.321608 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:22:26.321608 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:22:26.321608 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:22:26.321608 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:22:26.321608 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:22:26.321608 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:22:26.321608 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:22:26.321608 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:22:26.321608 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 04:22:26.321608 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 04:22:26.321608 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 04:22:26.321608 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 04:22:26.322516 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 04:22:26.322516 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 04:22:26.322516 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 04:22:26.322516 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 04:22:26.322516 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:22:26.322516 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:22:26.322516 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:22:26.322516 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:22:26.322516 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:22:26.322516 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:22:26.322516 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:22:26.322516 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:22:26.322516 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:22:26.322516 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:22:26.322516 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:22:26.322516 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:22:26.323320 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:22:26.323320 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:22:26.323320 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:22:26.323320 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:22:26.323320 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 04:22:26.323320 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 04:22:26.323320 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 04:22:26.323320 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 04:22:26.323320 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 04:22:26.323320 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 04:22:26.323320 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 25 04:22:26.323320 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:22:26.323320 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 169/327] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Sep 25 04:22:26 2022] GET /identity/v3/roles/reader => generated 84 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:22:26.331238 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24ffd0fb-3279-4625-ae77-7a1686ea92ae None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:26.394480 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:26.444023 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24ffd0fb-3279-4625-ae77-7a1686ea92ae None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:26.486636 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9731f65-563b-4018-8de7-cdd741306585 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:26.487576 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e80ee3e8-bd84-401d-9ce5-48f5390d0803 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:26.488196 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e80ee3e8-bd84-401d-9ce5-48f5390d0803 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:26.488476 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e80ee3e8-bd84-401d-9ce5-48f5390d0803 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:26.488844 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e80ee3e8-bd84-401d-9ce5-48f5390d0803 None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:26.489515 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e80ee3e8-bd84-401d-9ce5-48f5390d0803 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:26.489818 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e80ee3e8-bd84-401d-9ce5-48f5390d0803 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:26.491263 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e80ee3e8-bd84-401d-9ce5-48f5390d0803 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:26.500163 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 159/328] 10.222.0.25 () {56 vars in 1101 bytes} [Sun Sep 25 04:22:26 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:26.505990 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e80ee3e8-bd84-401d-9ce5-48f5390d0803 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:26.621335 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9731f65-563b-4018-8de7-cdd741306585 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:26.645970 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e80ee3e8-bd84-401d-9ce5-48f5390d0803 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:26.663837 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2904d15-15a6-48dc-b6a9-88f0de3460ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:26.664533 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2904d15-15a6-48dc-b6a9-88f0de3460ed None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:26.665060 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:26.665720 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2904d15-15a6-48dc-b6a9-88f0de3460ed None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:26.665891 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2904d15-15a6-48dc-b6a9-88f0de3460ed None admin] PATH_INFO: `/v3/roles/member` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:26.679402 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2904d15-15a6-48dc-b6a9-88f0de3460ed None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 04:22:26.684889 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b06c63a-d31b-42d3-91b9-6545e17c5032 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:26.685121 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-a2904d15-15a6-48dc-b6a9-88f0de3460ed None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 25 04:22:26.685121 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:22:26.685121 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:22:26.685121 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:22:26.685121 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:22:26.685121 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:22:26.685121 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:22:26.685121 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:22:26.685121 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:22:26.685121 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:22:26.685121 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:22:26.685121 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:22:26.685121 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 04:22:26.685121 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 04:22:26.685121 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 04:22:26.685121 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 04:22:26.685991 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 04:22:26.685991 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 04:22:26.685991 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 04:22:26.685991 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 04:22:26.685991 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:22:26.685991 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:22:26.685991 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:22:26.685991 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:22:26.685991 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:22:26.685991 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:22:26.685991 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:22:26.685991 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:22:26.685991 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:22:26.685991 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:22:26.685991 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:22:26.685991 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:22:26.686798 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:22:26.686798 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:22:26.686798 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:22:26.686798 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:22:26.686798 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 04:22:26.686798 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 04:22:26.686798 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 04:22:26.686798 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 04:22:26.686798 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 04:22:26.686798 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 04:22:26.686798 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 25 04:22:26.686798 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:22:26.687413 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b06c63a-d31b-42d3-91b9-6545e17c5032 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:26.687413 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b06c63a-d31b-42d3-91b9-6545e17c5032 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:26.687413 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b06c63a-d31b-42d3-91b9-6545e17c5032 None admin] PATH_INFO: `/v3/users/4d34992c3a9c4944a526d17d2b16acfb` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:26.687604 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 170/329] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Sep 25 04:22:26 2022] GET /identity/v3/roles/member => generated 84 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 04:22:26.689575 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b06c63a-d31b-42d3-91b9-6545e17c5032 None admin] RBAC: Authorizing `identity:get_user(user_id=4d34992c3a9c4944a526d17d2b16acfb)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:26.691107 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b06c63a-d31b-42d3-91b9-6545e17c5032 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:26.693485 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 160/330] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Sep 25 04:22:26 2022] GET /identity/v3/users/4d34992c3a9c4944a526d17d2b16acfb => generated 293 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:22:26.697626 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2904d15-15a6-48dc-b6a9-88f0de3460ed None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:26.700385 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b06c63a-d31b-42d3-91b9-6545e17c5032 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:26.755759 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:26.812630 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b06c63a-d31b-42d3-91b9-6545e17c5032 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:26.819206 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2904d15-15a6-48dc-b6a9-88f0de3460ed None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:26.853071 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36d290d8-4d1d-4308-8836-05d91ce8b3d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:26.853071 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d290d8-4d1d-4308-8836-05d91ce8b3d2 None admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:26.853333 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d290d8-4d1d-4308-8836-05d91ce8b3d2 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:26.853402 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d290d8-4d1d-4308-8836-05d91ce8b3d2 None admin] PATH_INFO: `/v3/system/users/4d34992c3a9c4944a526d17d2b16acfb/roles/0e4fa3b3402d463da90ddfcd17be7c17` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:26.854846 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36d290d8-4d1d-4308-8836-05d91ce8b3d2 None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=4d34992c3a9c4944a526d17d2b16acfb, role_id=0e4fa3b3402d463da90ddfcd17be7c17)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:26.856244 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36d290d8-4d1d-4308-8836-05d91ce8b3d2 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:26.867665 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c6472dc-dfbb-4237-b2fb-bcf4cfbd1776 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:26.868606 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c6472dc-dfbb-4237-b2fb-bcf4cfbd1776 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:26.868919 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 161/331] 10.222.0.25 () {58 vars in 1334 bytes} [Sun Sep 25 04:22:26 2022] PUT /identity/v3/system/users/4d34992c3a9c4944a526d17d2b16acfb/roles/0e4fa3b3402d463da90ddfcd17be7c17 => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 04:22:26.869330 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c6472dc-dfbb-4237-b2fb-bcf4cfbd1776 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:26.869816 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c6472dc-dfbb-4237-b2fb-bcf4cfbd1776 None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:26.870669 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c6472dc-dfbb-4237-b2fb-bcf4cfbd1776 None admin] RBAC: Adding query filter params (name=member) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:26.871164 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c6472dc-dfbb-4237-b2fb-bcf4cfbd1776 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:26.873991 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c6472dc-dfbb-4237-b2fb-bcf4cfbd1776 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:26.886435 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 171/332] 10.222.0.25 () {56 vars in 1101 bytes} [Sun Sep 25 04:22:26 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:26.893603 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36d290d8-4d1d-4308-8836-05d91ce8b3d2 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:26.962726 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:27.028388 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36d290d8-4d1d-4308-8836-05d91ce8b3d2 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:27.067293 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a4c5aa4-e871-4f73-a0f3-b6edcd3f3dfa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:27.068063 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a4c5aa4-e871-4f73-a0f3-b6edcd3f3dfa None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:27.068357 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a4c5aa4-e871-4f73-a0f3-b6edcd3f3dfa None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:27.068631 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a4c5aa4-e871-4f73-a0f3-b6edcd3f3dfa None admin] PATH_INFO: `/v3/users/3b93d7dd85bf4030a9d909bcc5613fcb` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:27.069899 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a4c5aa4-e871-4f73-a0f3-b6edcd3f3dfa None admin] RBAC: Authorizing `identity:get_user(user_id=3b93d7dd85bf4030a9d909bcc5613fcb)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:27.071487 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a4c5aa4-e871-4f73-a0f3-b6edcd3f3dfa None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:27.073306 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 162/333] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Sep 25 04:22:26 2022] GET /identity/v3/users/3b93d7dd85bf4030a9d909bcc5613fcb => generated 293 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:27.082559 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c6472dc-dfbb-4237-b2fb-bcf4cfbd1776 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:27.198708 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c6472dc-dfbb-4237-b2fb-bcf4cfbd1776 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:27.206697 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:27.246169 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de3bf5b2-43ed-4d11-aa27-cafac8fb1432 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:27.246789 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de3bf5b2-43ed-4d11-aa27-cafac8fb1432 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:27.247057 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de3bf5b2-43ed-4d11-aa27-cafac8fb1432 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:27.247331 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de3bf5b2-43ed-4d11-aa27-cafac8fb1432 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:27.248069 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de3bf5b2-43ed-4d11-aa27-cafac8fb1432 None admin] RBAC: Adding query filter params (role.id=8f6cc574056c4278bc9778f14dc4b411, user.id=3b93d7dd85bf4030a9d909bcc5613fcb, scope.system=all) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:27.248384 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de3bf5b2-43ed-4d11-aa27-cafac8fb1432 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:27.249890 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de3bf5b2-43ed-4d11-aa27-cafac8fb1432 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:27.268515 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 172/334] 10.222.0.25 () {56 vars in 1308 bytes} [Sun Sep 25 04:22:27 2022] GET /identity/v3/role_assignments?user.id=3b93d7dd85bf4030a9d909bcc5613fcb&scope.system=all&role.id=8f6cc574056c4278bc9778f14dc4b411 => generated 510 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:28.815393 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a410577-36d8-4d01-974c-96e810ea9d27 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:28.815913 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a410577-36d8-4d01-974c-96e810ea9d27 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:28.815913 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a410577-36d8-4d01-974c-96e810ea9d27 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:28.817540 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 163/335] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:22:28 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:22:28.823010 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c699247-aee2-4b75-b8ca-b8b355fbb143 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:28.823456 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c699247-aee2-4b75-b8ca-b8b355fbb143 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:28.823532 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c699247-aee2-4b75-b8ca-b8b355fbb143 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:28.843345 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-5c699247-aee2-4b75-b8ca-b8b355fbb143 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:22:28.845125 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-5c699247-aee2-4b75-b8ca-b8b355fbb143 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:22:28.927364 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 173/336] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:22:28 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 04:22:29.201778 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f132dce-9e3b-4cbe-84a4-81e7e552a807 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:29.202156 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f132dce-9e3b-4cbe-84a4-81e7e552a807 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:29.202367 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f132dce-9e3b-4cbe-84a4-81e7e552a807 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:29.204173 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 164/337] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:22:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:22:29.209191 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf0e08f-d198-4692-a47e-5cd768698c69 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:29.209476 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf0e08f-d198-4692-a47e-5cd768698c69 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:29.209750 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf0e08f-d198-4692-a47e-5cd768698c69 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:29.233034 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-bdf0e08f-d198-4692-a47e-5cd768698c69 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:22:29.234221 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-bdf0e08f-d198-4692-a47e-5cd768698c69 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:22:29.310468 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 174/338] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:22:29 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 04:22:29.482375 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f132dce-9e3b-4cbe-84a4-81e7e552a807 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:29.618569 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f132dce-9e3b-4cbe-84a4-81e7e552a807 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:29.638091 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:29.659954 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ec4a97c-40be-451b-bd33-d286cd32e611 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:29.660593 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec4a97c-40be-451b-bd33-d286cd32e611 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:29.660905 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec4a97c-40be-451b-bd33-d286cd32e611 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:29.661180 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec4a97c-40be-451b-bd33-d286cd32e611 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:29.672399 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ec4a97c-40be-451b-bd33-d286cd32e611 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 04:22:29.677791 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-8ec4a97c-40be-451b-bd33-d286cd32e611 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 25 04:22:29.677791 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:22:29.677791 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:22:29.677791 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:22:29.677791 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:22:29.677791 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:22:29.677791 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:22:29.677791 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:22:29.677791 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:22:29.677791 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:22:29.677791 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:22:29.677791 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:22:29.677791 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 04:22:29.677791 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 04:22:29.677791 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 04:22:29.677791 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 04:22:29.678691 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 04:22:29.678691 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 04:22:29.678691 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 04:22:29.678691 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 04:22:29.678691 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:22:29.678691 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:22:29.678691 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:22:29.678691 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:22:29.678691 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:22:29.678691 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:22:29.678691 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:22:29.678691 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:22:29.678691 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:22:29.678691 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:22:29.678691 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:22:29.678691 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:22:29.679498 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:22:29.679498 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:22:29.679498 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:22:29.679498 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:22:29.679498 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 04:22:29.679498 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 04:22:29.679498 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 04:22:29.679498 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 04:22:29.679498 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 04:22:29.679498 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 04:22:29.679498 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 25 04:22:29.679498 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:22:29.679498 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 165/339] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Sep 25 04:22:29 2022] GET /identity/v3/roles/member => generated 84 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:22:29.684749 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdf0e08f-d198-4692-a47e-5cd768698c69 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:29.731408 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:29.828313 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ec4a97c-40be-451b-bd33-d286cd32e611 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:29.851510 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdf0e08f-d198-4692-a47e-5cd768698c69 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:29.915196 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e71b3ff3-d3f0-401f-b598-524c503d4fe7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:29.915813 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e71b3ff3-d3f0-401f-b598-524c503d4fe7 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:29.916091 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e71b3ff3-d3f0-401f-b598-524c503d4fe7 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:29.916357 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e71b3ff3-d3f0-401f-b598-524c503d4fe7 None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:29.917049 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e71b3ff3-d3f0-401f-b598-524c503d4fe7 None admin] RBAC: Adding query filter params (name=member) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:29.917349 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e71b3ff3-d3f0-401f-b598-524c503d4fe7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:29.918713 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e71b3ff3-d3f0-401f-b598-524c503d4fe7 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:29.942505 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 175/340] 10.222.0.25 () {56 vars in 1101 bytes} [Sun Sep 25 04:22:29 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 258 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:29.944630 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f44ab6-0be4-4110-a214-1e71f35ce1d9 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:29.945267 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f44ab6-0be4-4110-a214-1e71f35ce1d9 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:29.945740 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f44ab6-0be4-4110-a214-1e71f35ce1d9 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:29.947279 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 176/341] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:22:29 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:22:29.951198 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47f44ab6-0be4-4110-a214-1e71f35ce1d9 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:30.036880 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ec4a97c-40be-451b-bd33-d286cd32e611 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:30.080952 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6d66f97-06e3-4df5-b98f-eeb89d526c7d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:30.081600 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d66f97-06e3-4df5-b98f-eeb89d526c7d None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:30.081887 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d66f97-06e3-4df5-b98f-eeb89d526c7d None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:30.082160 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d66f97-06e3-4df5-b98f-eeb89d526c7d None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:30.091060 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:30.103761 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6d66f97-06e3-4df5-b98f-eeb89d526c7d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 04:22:30.109800 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-e6d66f97-06e3-4df5-b98f-eeb89d526c7d None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 25 04:22:30.109800 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:22:30.109800 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:22:30.109800 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:22:30.109800 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:22:30.109800 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:22:30.109800 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:22:30.109800 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:22:30.109800 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:22:30.109800 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:22:30.109800 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:22:30.109800 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:22:30.109800 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 04:22:30.109800 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 04:22:30.109800 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 04:22:30.109800 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 04:22:30.110896 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 04:22:30.110896 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 04:22:30.110896 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 04:22:30.110896 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 04:22:30.110896 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:22:30.110896 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:22:30.110896 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:22:30.110896 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:22:30.110896 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:22:30.110896 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:22:30.110896 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:22:30.110896 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:22:30.110896 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:22:30.110896 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:22:30.110896 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:22:30.110896 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:22:30.112111 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:22:30.112111 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:22:30.112111 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:22:30.112111 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:22:30.112111 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 04:22:30.112111 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 04:22:30.112111 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 04:22:30.112111 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 04:22:30.112111 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 04:22:30.112111 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 04:22:30.112111 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 25 04:22:30.112111 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:22:30.112111 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 166/342] 10.222.0.25 () {56 vars in 1114 bytes} [Sun Sep 25 04:22:29 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 284 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 04:22:30.113090 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b0ee851-bdf9-4dde-80d8-1584cb811bf7 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:30.113090 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b0ee851-bdf9-4dde-80d8-1584cb811bf7 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:30.119809 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b0ee851-bdf9-4dde-80d8-1584cb811bf7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:30.129034 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47f44ab6-0be4-4110-a214-1e71f35ce1d9 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:30.164469 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-6b0ee851-bdf9-4dde-80d8-1584cb811bf7 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:22:30.165731 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-6b0ee851-bdf9-4dde-80d8-1584cb811bf7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:22:30.195338 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-074a7cbf-395f-43be-be13-ea897e27ffb4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:30.195967 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-074a7cbf-395f-43be-be13-ea897e27ffb4 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:30.196242 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-074a7cbf-395f-43be-be13-ea897e27ffb4 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:30.196510 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-074a7cbf-395f-43be-be13-ea897e27ffb4 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:30.201776 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:30.274015 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 167/343] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:22:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 04:22:30.274908 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b0ee851-bdf9-4dde-80d8-1584cb811bf7 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:30.290103 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-074a7cbf-395f-43be-be13-ea897e27ffb4 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:30.292149 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-074a7cbf-395f-43be-be13-ea897e27ffb4 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:30.403457 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:30.424924 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 177/344] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Sep 25 04:22:29 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 474 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 25 04:22:30.430794 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-074a7cbf-395f-43be-be13-ea897e27ffb4 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:30.438361 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b0ee851-bdf9-4dde-80d8-1584cb811bf7 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:30.482454 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca7a976f-93f3-4c21-9931-8434183b8978 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:30.483440 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca7a976f-93f3-4c21-9931-8434183b8978 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:30.483901 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca7a976f-93f3-4c21-9931-8434183b8978 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:30.484365 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca7a976f-93f3-4c21-9931-8434183b8978 None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:30.485609 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca7a976f-93f3-4c21-9931-8434183b8978 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:30.486378 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca7a976f-93f3-4c21-9931-8434183b8978 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:30.488496 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca7a976f-93f3-4c21-9931-8434183b8978 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:30.498765 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 168/345] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Sep 25 04:22:30 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:30.505561 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca7a976f-93f3-4c21-9931-8434183b8978 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:30.539795 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:30.570381 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-074a7cbf-395f-43be-be13-ea897e27ffb4 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:30.616447 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1f9b9b9-74a2-4b8a-9c11-e853166a7bf4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:30.617123 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f9b9b9-74a2-4b8a-9c11-e853166a7bf4 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:30.617400 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f9b9b9-74a2-4b8a-9c11-e853166a7bf4 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:30.617671 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f9b9b9-74a2-4b8a-9c11-e853166a7bf4 None admin] PATH_INFO: `/v3/projects/bceb73400ebb4062be126679d2f16f6d` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:30.619087 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1f9b9b9-74a2-4b8a-9c11-e853166a7bf4 None admin] RBAC: Authorizing `identity:get_project(project_id=bceb73400ebb4062be126679d2f16f6d)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:30.620911 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1f9b9b9-74a2-4b8a-9c11-e853166a7bf4 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:30.623062 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 178/346] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Sep 25 04:22:30 2022] GET /identity/v3/projects/bceb73400ebb4062be126679d2f16f6d => generated 298 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:22:30.629311 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1f9b9b9-74a2-4b8a-9c11-e853166a7bf4 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:30.629890 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca7a976f-93f3-4c21-9931-8434183b8978 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:30.666318 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfcbaa50-438f-43aa-914e-cc5ef857bdb6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:30.667041 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfcbaa50-438f-43aa-914e-cc5ef857bdb6 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:30.667395 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfcbaa50-438f-43aa-914e-cc5ef857bdb6 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:30.667742 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfcbaa50-438f-43aa-914e-cc5ef857bdb6 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:30.728220 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:30.752519 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfcbaa50-438f-43aa-914e-cc5ef857bdb6 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:30.754037 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfcbaa50-438f-43aa-914e-cc5ef857bdb6 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:30.789633 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:30.811687 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1f9b9b9-74a2-4b8a-9c11-e853166a7bf4 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:30.864271 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c154f96-cd65-450e-9f69-94f6b3f6cabf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:30.865704 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c154f96-cd65-450e-9f69-94f6b3f6cabf None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:30.870413 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c154f96-cd65-450e-9f69-94f6b3f6cabf None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:30.870950 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c154f96-cd65-450e-9f69-94f6b3f6cabf None admin] PATH_INFO: `/v3/groups/1f61c67adfcd46cfadf3d92d2ae29f7a` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:30.873915 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 169/347] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Sep 25 04:22:30 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 369 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 25 04:22:30.875215 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cd7c9cd-c2d9-4178-9162-7f0202c73a73 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:30.875501 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cd7c9cd-c2d9-4178-9162-7f0202c73a73 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:30.875908 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cd7c9cd-c2d9-4178-9162-7f0202c73a73 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:30.879279 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c154f96-cd65-450e-9f69-94f6b3f6cabf None admin] RBAC: Authorizing `identity:get_group(group_id=1f61c67adfcd46cfadf3d92d2ae29f7a)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:30.880973 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c154f96-cd65-450e-9f69-94f6b3f6cabf None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:30.881747 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 170/348] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:22:30 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:22:30.883286 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 179/349] 10.222.0.25 () {56 vars in 1180 bytes} [Sun Sep 25 04:22:30 2022] GET /identity/v3/groups/1f61c67adfcd46cfadf3d92d2ae29f7a => generated 226 bytes in 254 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:30.884750 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cd7c9cd-c2d9-4178-9162-7f0202c73a73 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:30.890850 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c154f96-cd65-450e-9f69-94f6b3f6cabf None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:31.009032 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cd7c9cd-c2d9-4178-9162-7f0202c73a73 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:31.027553 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c154f96-cd65-450e-9f69-94f6b3f6cabf None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:31.049285 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:31.052039 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:31.061822 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-468d440d-ef41-4623-9704-9a220ef1da81 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:31.062850 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-468d440d-ef41-4623-9704-9a220ef1da81 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:31.063384 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-468d440d-ef41-4623-9704-9a220ef1da81 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:31.063873 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-468d440d-ef41-4623-9704-9a220ef1da81 None admin] PATH_INFO: `/v3/projects/bceb73400ebb4062be126679d2f16f6d` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:31.065352 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-468d440d-ef41-4623-9704-9a220ef1da81 None admin] RBAC: Authorizing `identity:get_project(project_id=bceb73400ebb4062be126679d2f16f6d)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:31.066932 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-468d440d-ef41-4623-9704-9a220ef1da81 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:31.072660 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 171/350] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Sep 25 04:22:30 2022] GET /identity/v3/projects/bceb73400ebb4062be126679d2f16f6d => generated 298 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:22:31.072660 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-767bb2ed-1385-475a-b8d9-b9ae27f12020 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:31.072660 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-767bb2ed-1385-475a-b8d9-b9ae27f12020 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:31.072660 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-767bb2ed-1385-475a-b8d9-b9ae27f12020 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:31.084239 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8afef3c-b3a8-4d61-95a7-2bc339cec3e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:31.084912 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8afef3c-b3a8-4d61-95a7-2bc339cec3e8 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:31.085761 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8afef3c-b3a8-4d61-95a7-2bc339cec3e8 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:31.086043 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8afef3c-b3a8-4d61-95a7-2bc339cec3e8 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:31.086786 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8afef3c-b3a8-4d61-95a7-2bc339cec3e8 None admin] RBAC: Adding query filter params (group.id=1f61c67adfcd46cfadf3d92d2ae29f7a, role.id=8f6cc574056c4278bc9778f14dc4b411, scope.project.id=bceb73400ebb4062be126679d2f16f6d) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:31.087097 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8afef3c-b3a8-4d61-95a7-2bc339cec3e8 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:31.089344 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8afef3c-b3a8-4d61-95a7-2bc339cec3e8 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:31.092163 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-767bb2ed-1385-475a-b8d9-b9ae27f12020 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:22:31.093441 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-767bb2ed-1385-475a-b8d9-b9ae27f12020 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:22:31.099240 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 180/351] 10.222.0.25 () {56 vars in 1376 bytes} [Sun Sep 25 04:22:30 2022] GET /identity/v3/role_assignments?group.id=1f61c67adfcd46cfadf3d92d2ae29f7a&scope.project.id=bceb73400ebb4062be126679d2f16f6d&role.id=8f6cc574056c4278bc9778f14dc4b411 => generated 260 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:31.101360 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15684c08-78e2-480d-a6a4-01d5e5090de7 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:31.101714 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15684c08-78e2-480d-a6a4-01d5e5090de7 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:31.101996 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15684c08-78e2-480d-a6a4-01d5e5090de7 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:31.103315 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 181/352] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:22:31 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:22:31.104048 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15684c08-78e2-480d-a6a4-01d5e5090de7 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:31.217868 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 172/353] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:22:31 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 04:22:31.220414 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-767bb2ed-1385-475a-b8d9-b9ae27f12020 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:31.261167 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15684c08-78e2-480d-a6a4-01d5e5090de7 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:31.332991 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-480e800f-4300-4889-8131-fb8a316dd4f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:31.334002 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-480e800f-4300-4889-8131-fb8a316dd4f3 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:31.334484 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-480e800f-4300-4889-8131-fb8a316dd4f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:31.334977 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-480e800f-4300-4889-8131-fb8a316dd4f3 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:31.359651 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-480e800f-4300-4889-8131-fb8a316dd4f3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 04:22:31.366785 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-480e800f-4300-4889-8131-fb8a316dd4f3 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 25 04:22:31.366785 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:22:31.366785 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:22:31.366785 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:22:31.366785 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:22:31.366785 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:22:31.366785 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:22:31.366785 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:22:31.366785 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:22:31.366785 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:22:31.366785 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:22:31.366785 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:22:31.366785 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 04:22:31.366785 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 04:22:31.366785 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 04:22:31.366785 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 04:22:31.367788 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 04:22:31.367788 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 04:22:31.367788 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 04:22:31.367788 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 04:22:31.367788 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:22:31.367788 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:22:31.367788 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:22:31.367788 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:22:31.367788 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:22:31.367788 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:22:31.367788 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:22:31.367788 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:22:31.367788 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:22:31.367788 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:22:31.367788 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:22:31.367788 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:22:31.368684 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:22:31.368684 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:22:31.368684 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:22:31.368684 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:22:31.368684 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 04:22:31.368684 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 04:22:31.368684 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 04:22:31.368684 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 04:22:31.368684 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 04:22:31.368684 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 04:22:31.368684 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 25 04:22:31.368684 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:22:31.379351 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 182/354] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Sep 25 04:22:31 2022] GET /identity/v3/roles/member => generated 84 bytes in 275 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:22:31.380943 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f9d52ee-c484-444f-99bf-4e16d164de77 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:31.383653 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f9d52ee-c484-444f-99bf-4e16d164de77 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:31.384149 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f9d52ee-c484-444f-99bf-4e16d164de77 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:31.413105 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-2f9d52ee-c484-444f-99bf-4e16d164de77 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:22:31.414381 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-2f9d52ee-c484-444f-99bf-4e16d164de77 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:22:31.428970 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-767bb2ed-1385-475a-b8d9-b9ae27f12020 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:31.464350 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:31.502524 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:31.506557 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 183/355] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:22:31 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 04:22:31.507313 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f9d52ee-c484-444f-99bf-4e16d164de77 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:31.537382 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7b3f95b-85e6-451f-90cd-a89abc28b6e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:31.538049 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b3f95b-85e6-451f-90cd-a89abc28b6e8 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:31.538333 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b3f95b-85e6-451f-90cd-a89abc28b6e8 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:31.538608 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b3f95b-85e6-451f-90cd-a89abc28b6e8 None admin] PATH_INFO: `/v3/groups/1f61c67adfcd46cfadf3d92d2ae29f7a` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:31.539779 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7b3f95b-85e6-451f-90cd-a89abc28b6e8 None admin] RBAC: Authorizing `identity:get_group(group_id=1f61c67adfcd46cfadf3d92d2ae29f7a)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:31.541264 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7b3f95b-85e6-451f-90cd-a89abc28b6e8 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:31.543152 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 173/356] 10.222.0.25 () {56 vars in 1180 bytes} [Sun Sep 25 04:22:31 2022] GET /identity/v3/groups/1f61c67adfcd46cfadf3d92d2ae29f7a => generated 226 bytes in 323 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:31.549392 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7b3f95b-85e6-451f-90cd-a89abc28b6e8 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:31.627406 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f9d52ee-c484-444f-99bf-4e16d164de77 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:31.670423 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8ebdafd-77f7-4bc5-9aaf-473ce5c84742 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:31.671044 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ebdafd-77f7-4bc5-9aaf-473ce5c84742 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:31.671594 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ebdafd-77f7-4bc5-9aaf-473ce5c84742 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:31.671871 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ebdafd-77f7-4bc5-9aaf-473ce5c84742 None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:31.672572 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8ebdafd-77f7-4bc5-9aaf-473ce5c84742 None admin] RBAC: Adding query filter params (name=member) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:31.673371 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8ebdafd-77f7-4bc5-9aaf-473ce5c84742 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:31.674773 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8ebdafd-77f7-4bc5-9aaf-473ce5c84742 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:31.684748 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 184/357] 10.222.0.25 () {56 vars in 1101 bytes} [Sun Sep 25 04:22:31 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:31.685581 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8ebdafd-77f7-4bc5-9aaf-473ce5c84742 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:31.726198 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:31.770103 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7b3f95b-85e6-451f-90cd-a89abc28b6e8 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:31.806498 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8ebdafd-77f7-4bc5-9aaf-473ce5c84742 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:31.809316 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90650a03-c071-4bc8-84e6-b154eaa9ad8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:31.809937 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90650a03-c071-4bc8-84e6-b154eaa9ad8c None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:31.810216 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90650a03-c071-4bc8-84e6-b154eaa9ad8c None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:31.810489 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90650a03-c071-4bc8-84e6-b154eaa9ad8c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:31.811217 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90650a03-c071-4bc8-84e6-b154eaa9ad8c None admin] RBAC: Adding query filter params (group.id=1f61c67adfcd46cfadf3d92d2ae29f7a, role.id=02a8d263519b4ff1b511a8577293126c, scope.project.id=bceb73400ebb4062be126679d2f16f6d) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:31.811527 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90650a03-c071-4bc8-84e6-b154eaa9ad8c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:31.813019 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90650a03-c071-4bc8-84e6-b154eaa9ad8c None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:31.819214 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 174/358] 10.222.0.25 () {56 vars in 1376 bytes} [Sun Sep 25 04:22:31 2022] GET /identity/v3/role_assignments?group.id=1f61c67adfcd46cfadf3d92d2ae29f7a&scope.project.id=bceb73400ebb4062be126679d2f16f6d&role.id=02a8d263519b4ff1b511a8577293126c => generated 260 bytes in 270 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:31.820019 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90650a03-c071-4bc8-84e6-b154eaa9ad8c None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:31.846016 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3b5ffad-caad-4acf-90ed-6ec5fc432960 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:31.846665 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3b5ffad-caad-4acf-90ed-6ec5fc432960 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:31.846946 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3b5ffad-caad-4acf-90ed-6ec5fc432960 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:31.847220 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3b5ffad-caad-4acf-90ed-6ec5fc432960 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:31.859833 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3b5ffad-caad-4acf-90ed-6ec5fc432960 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 04:22:31.866626 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-f3b5ffad-caad-4acf-90ed-6ec5fc432960 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 25 04:22:31.866626 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:22:31.866626 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:22:31.866626 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:22:31.866626 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:22:31.866626 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:22:31.866626 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:22:31.866626 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:22:31.866626 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:22:31.866626 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:22:31.866626 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:22:31.866626 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:22:31.866626 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 04:22:31.866626 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 04:22:31.866626 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 04:22:31.866626 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 04:22:31.868051 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 04:22:31.868051 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 04:22:31.868051 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 04:22:31.868051 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 04:22:31.868051 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:22:31.868051 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:22:31.868051 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:22:31.868051 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:22:31.868051 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:22:31.868051 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:22:31.868051 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:22:31.868051 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:22:31.868051 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:22:31.868051 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:22:31.868051 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:22:31.868051 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:22:31.868914 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:22:31.868914 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:22:31.868914 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:22:31.868914 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:22:31.868914 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 04:22:31.868914 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 04:22:31.868914 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 04:22:31.868914 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 04:22:31.868914 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 04:22:31.868914 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 04:22:31.868914 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 25 04:22:31.868914 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:22:31.868914 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 185/359] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Sep 25 04:22:31 2022] GET /identity/v3/roles/reader => generated 84 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:22:31.872035 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9261bcf3-9d95-4b22-9015-b1b56e276299 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:31.872035 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9261bcf3-9d95-4b22-9015-b1b56e276299 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:31.872035 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9261bcf3-9d95-4b22-9015-b1b56e276299 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:31.873219 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 186/360] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:22:31 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:22:31.875409 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9261bcf3-9d95-4b22-9015-b1b56e276299 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:31.892406 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:31.943082 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90650a03-c071-4bc8-84e6-b154eaa9ad8c None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:31.993408 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:32.002843 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb230619-4d60-4173-b1f5-317f2cfce26e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:32.003539 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb230619-4d60-4173-b1f5-317f2cfce26e None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:32.003826 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb230619-4d60-4173-b1f5-317f2cfce26e None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:32.004097 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb230619-4d60-4173-b1f5-317f2cfce26e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:32.004203 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9261bcf3-9d95-4b22-9015-b1b56e276299 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:32.049193 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48b73f15-7068-4939-bf29-b4d6b8b8b9bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:32.049729 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48b73f15-7068-4939-bf29-b4d6b8b8b9bf None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:32.050010 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48b73f15-7068-4939-bf29-b4d6b8b8b9bf None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:32.050449 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48b73f15-7068-4939-bf29-b4d6b8b8b9bf None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:32.051560 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48b73f15-7068-4939-bf29-b4d6b8b8b9bf None admin] RBAC: Adding query filter params (name=reader) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:32.052141 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48b73f15-7068-4939-bf29-b4d6b8b8b9bf None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:32.053667 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48b73f15-7068-4939-bf29-b4d6b8b8b9bf None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:32.066871 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 187/361] 10.222.0.25 () {56 vars in 1101 bytes} [Sun Sep 25 04:22:31 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:32.068155 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40d15862-b228-4a62-af18-ad37e8077feb None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:32.068438 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40d15862-b228-4a62-af18-ad37e8077feb None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:32.072779 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40d15862-b228-4a62-af18-ad37e8077feb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:32.097476 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-40d15862-b228-4a62-af18-ad37e8077feb None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:22:32.098684 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-40d15862-b228-4a62-af18-ad37e8077feb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:22:32.104270 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb230619-4d60-4173-b1f5-317f2cfce26e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:32.106798 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb230619-4d60-4173-b1f5-317f2cfce26e None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:32.169128 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:32.174280 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 188/362] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:22:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 04:22:32.175270 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40d15862-b228-4a62-af18-ad37e8077feb None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:32.216160 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 175/363] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Sep 25 04:22:31 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 396 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 25 04:22:32.218004 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb230619-4d60-4173-b1f5-317f2cfce26e None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:32.269544 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:32.288399 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40d15862-b228-4a62-af18-ad37e8077feb None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:32.327800 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0a199fb-fd80-46e1-ace6-4f1a3d7ecebe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:32.329237 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0a199fb-fd80-46e1-ace6-4f1a3d7ecebe None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:32.329688 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0a199fb-fd80-46e1-ace6-4f1a3d7ecebe None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:32.331406 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0a199fb-fd80-46e1-ace6-4f1a3d7ecebe None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:32.331534 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb230619-4d60-4173-b1f5-317f2cfce26e None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:32.342557 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0a199fb-fd80-46e1-ace6-4f1a3d7ecebe None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 04:22:32.347878 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-e0a199fb-fd80-46e1-ace6-4f1a3d7ecebe None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 25 04:22:32.347878 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:22:32.347878 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:22:32.347878 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:22:32.347878 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:22:32.347878 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:22:32.347878 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:22:32.347878 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:22:32.347878 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:22:32.347878 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:22:32.347878 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:22:32.347878 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:22:32.347878 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 04:22:32.347878 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 04:22:32.347878 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 04:22:32.347878 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 04:22:32.349963 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 04:22:32.349963 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 04:22:32.349963 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 04:22:32.349963 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 04:22:32.349963 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:22:32.349963 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:22:32.349963 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:22:32.349963 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:22:32.349963 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:22:32.349963 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:22:32.349963 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:22:32.349963 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:22:32.349963 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:22:32.349963 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:22:32.349963 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:22:32.349963 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:22:32.350786 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:22:32.350786 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:22:32.350786 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:22:32.350786 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:22:32.350786 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 04:22:32.350786 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 04:22:32.350786 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 04:22:32.350786 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 04:22:32.350786 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 04:22:32.350786 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 04:22:32.350786 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 25 04:22:32.350786 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:22:32.352982 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 189/364] 10.222.0.25 () {56 vars in 1114 bytes} [Sun Sep 25 04:22:32 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:22:32.354115 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0a199fb-fd80-46e1-ace6-4f1a3d7ecebe None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:32.370317 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-721339e0-5f78-4b08-961f-ecea44fd7fe5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:32.370962 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-721339e0-5f78-4b08-961f-ecea44fd7fe5 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:32.371245 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-721339e0-5f78-4b08-961f-ecea44fd7fe5 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:32.371519 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-721339e0-5f78-4b08-961f-ecea44fd7fe5 None admin] PATH_INFO: `/v3/users/4d34992c3a9c4944a526d17d2b16acfb` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:32.373519 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-721339e0-5f78-4b08-961f-ecea44fd7fe5 None admin] RBAC: Authorizing `identity:get_user(user_id=4d34992c3a9c4944a526d17d2b16acfb)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:32.374973 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-721339e0-5f78-4b08-961f-ecea44fd7fe5 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:32.376976 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 176/365] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Sep 25 04:22:32 2022] GET /identity/v3/users/4d34992c3a9c4944a526d17d2b16acfb => generated 293 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:32.377747 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-721339e0-5f78-4b08-961f-ecea44fd7fe5 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:32.417188 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:32.479810 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0a199fb-fd80-46e1-ace6-4f1a3d7ecebe None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:32.524065 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-721339e0-5f78-4b08-961f-ecea44fd7fe5 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:32.552347 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d4c58e2-1d12-4f33-a803-eca8a41ca865 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:32.553071 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d4c58e2-1d12-4f33-a803-eca8a41ca865 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:32.553352 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d4c58e2-1d12-4f33-a803-eca8a41ca865 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:32.553623 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d4c58e2-1d12-4f33-a803-eca8a41ca865 None admin] PATH_INFO: `/v3/projects/12400184cf3844998e5e6d8078b8cda4` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:32.554889 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d4c58e2-1d12-4f33-a803-eca8a41ca865 None admin] RBAC: Authorizing `identity:get_project(project_id=12400184cf3844998e5e6d8078b8cda4)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:32.556411 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d4c58e2-1d12-4f33-a803-eca8a41ca865 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:32.562742 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 190/366] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Sep 25 04:22:32 2022] GET /identity/v3/projects/12400184cf3844998e5e6d8078b8cda4 => generated 302 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:32.563843 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d4c58e2-1d12-4f33-a803-eca8a41ca865 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:32.573279 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19275af0-df5a-4fb5-91e1-edf663451cca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:32.573896 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19275af0-df5a-4fb5-91e1-edf663451cca None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:32.574178 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19275af0-df5a-4fb5-91e1-edf663451cca None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:32.574449 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19275af0-df5a-4fb5-91e1-edf663451cca None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:32.575114 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19275af0-df5a-4fb5-91e1-edf663451cca None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:32.575675 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19275af0-df5a-4fb5-91e1-edf663451cca None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:32.577318 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19275af0-df5a-4fb5-91e1-edf663451cca None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:32.586258 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 177/367] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Sep 25 04:22:32 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:32.587019 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19275af0-df5a-4fb5-91e1-edf663451cca None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:32.591172 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:32.682569 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:32.697244 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19275af0-df5a-4fb5-91e1-edf663451cca None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:32.716102 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d4c58e2-1d12-4f33-a803-eca8a41ca865 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:32.734962 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37dca447-1780-49a8-a6cf-2b376d770b79 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:32.735937 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37dca447-1780-49a8-a6cf-2b376d770b79 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:32.735937 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37dca447-1780-49a8-a6cf-2b376d770b79 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:32.736129 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37dca447-1780-49a8-a6cf-2b376d770b79 None admin] PATH_INFO: `/v3/groups/1f61c67adfcd46cfadf3d92d2ae29f7a` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:32.737283 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37dca447-1780-49a8-a6cf-2b376d770b79 None admin] RBAC: Authorizing `identity:get_group(group_id=1f61c67adfcd46cfadf3d92d2ae29f7a)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:32.738717 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37dca447-1780-49a8-a6cf-2b376d770b79 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:32.740603 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 178/368] 10.222.0.25 () {56 vars in 1180 bytes} [Sun Sep 25 04:22:32 2022] GET /identity/v3/groups/1f61c67adfcd46cfadf3d92d2ae29f7a => generated 226 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:32.741588 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37dca447-1780-49a8-a6cf-2b376d770b79 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:32.759418 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba9fd2a4-badc-4b03-9db8-f91106a122af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:32.760915 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba9fd2a4-badc-4b03-9db8-f91106a122af None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:32.761672 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba9fd2a4-badc-4b03-9db8-f91106a122af None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:32.762132 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba9fd2a4-badc-4b03-9db8-f91106a122af None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:32.763232 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba9fd2a4-badc-4b03-9db8-f91106a122af None admin] RBAC: Adding query filter params (role.id=0e4fa3b3402d463da90ddfcd17be7c17, user.id=4d34992c3a9c4944a526d17d2b16acfb, scope.system=all) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:32.763846 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba9fd2a4-badc-4b03-9db8-f91106a122af None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:32.765607 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba9fd2a4-badc-4b03-9db8-f91106a122af None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:32.781704 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 191/369] 10.222.0.25 () {56 vars in 1308 bytes} [Sun Sep 25 04:22:32 2022] GET /identity/v3/role_assignments?user.id=4d34992c3a9c4944a526d17d2b16acfb&scope.system=all&role.id=0e4fa3b3402d463da90ddfcd17be7c17 => generated 510 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:32.783398 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba9fd2a4-badc-4b03-9db8-f91106a122af None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:32.880059 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:32.907950 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba9fd2a4-badc-4b03-9db8-f91106a122af None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:32.950581 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dad3a315-6ee0-4403-8035-00a64f9898ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:32.951634 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad3a315-6ee0-4403-8035-00a64f9898ca None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:32.952112 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad3a315-6ee0-4403-8035-00a64f9898ca None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:32.952763 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad3a315-6ee0-4403-8035-00a64f9898ca None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:32.966379 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dad3a315-6ee0-4403-8035-00a64f9898ca None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 04:22:32.972625 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-dad3a315-6ee0-4403-8035-00a64f9898ca None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 04:22:32.972625 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:22:32.972625 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:22:32.972625 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:22:32.972625 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:22:32.972625 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:22:32.972625 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:22:32.972625 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:22:32.972625 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:22:32.972625 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:22:32.972625 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:22:32.972625 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:22:32.972625 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 04:22:32.972625 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 04:22:32.972625 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 04:22:32.972625 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 04:22:32.973613 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 04:22:32.973613 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 04:22:32.973613 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 04:22:32.973613 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 04:22:32.973613 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:22:32.973613 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:22:32.973613 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:22:32.973613 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:22:32.973613 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:22:32.973613 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:22:32.973613 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:22:32.973613 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:22:32.973613 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:22:32.973613 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:22:32.973613 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:22:32.973613 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:22:32.974436 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:22:32.974436 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:22:32.974436 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:22:32.974436 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:22:32.974436 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 04:22:32.974436 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 04:22:32.974436 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 04:22:32.974436 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 04:22:32.974436 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 04:22:32.974436 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 04:22:32.974436 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 04:22:32.974436 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:22:32.982603 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 192/370] 10.222.0.25 () {56 vars in 1096 bytes} [Sun Sep 25 04:22:32 2022] GET /identity/v3/roles/admin => generated 83 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:22:32.983978 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dad3a315-6ee0-4403-8035-00a64f9898ca None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:32.998906 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37dca447-1780-49a8-a6cf-2b376d770b79 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:33.079784 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f8ec1fb-cba0-4843-af73-18bd74681345 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:33.080460 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f8ec1fb-cba0-4843-af73-18bd74681345 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:33.080840 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f8ec1fb-cba0-4843-af73-18bd74681345 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:33.081120 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f8ec1fb-cba0-4843-af73-18bd74681345 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:33.109258 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dad3a315-6ee0-4403-8035-00a64f9898ca None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:33.119492 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:33.159632 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-038cc3d1-06b2-49e5-bcaf-193148b17e37 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:33.160784 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-038cc3d1-06b2-49e5-bcaf-193148b17e37 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:33.161402 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-038cc3d1-06b2-49e5-bcaf-193148b17e37 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:33.161910 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-038cc3d1-06b2-49e5-bcaf-193148b17e37 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:33.163041 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-038cc3d1-06b2-49e5-bcaf-193148b17e37 None admin] RBAC: Adding query filter params (group.id=1f61c67adfcd46cfadf3d92d2ae29f7a, role.id=8f6cc574056c4278bc9778f14dc4b411, scope.project.id=12400184cf3844998e5e6d8078b8cda4) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:33.163986 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-038cc3d1-06b2-49e5-bcaf-193148b17e37 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:33.166088 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-038cc3d1-06b2-49e5-bcaf-193148b17e37 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:33.175116 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 193/371] 10.222.0.25 () {56 vars in 1376 bytes} [Sun Sep 25 04:22:32 2022] GET /identity/v3/role_assignments?group.id=1f61c67adfcd46cfadf3d92d2ae29f7a&scope.project.id=12400184cf3844998e5e6d8078b8cda4&role.id=8f6cc574056c4278bc9778f14dc4b411 => generated 260 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:33.176440 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-038cc3d1-06b2-49e5-bcaf-193148b17e37 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:33.176562 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f8ec1fb-cba0-4843-af73-18bd74681345 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:33.177990 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f8ec1fb-cba0-4843-af73-18bd74681345 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:33.189099 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:33.372974 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 179/372] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Sep 25 04:22:32 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 632 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 25 04:22:33.381550 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f8ec1fb-cba0-4843-af73-18bd74681345 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:33.395699 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-038cc3d1-06b2-49e5-bcaf-193148b17e37 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:33.456936 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b7ad694-2d8e-4845-bd1d-214cc96ad247 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:33.457486 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b7ad694-2d8e-4845-bd1d-214cc96ad247 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:33.457769 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b7ad694-2d8e-4845-bd1d-214cc96ad247 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:33.458039 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b7ad694-2d8e-4845-bd1d-214cc96ad247 None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:33.458713 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b7ad694-2d8e-4845-bd1d-214cc96ad247 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:33.459017 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b7ad694-2d8e-4845-bd1d-214cc96ad247 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:33.460388 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b7ad694-2d8e-4845-bd1d-214cc96ad247 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:33.491067 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 194/373] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Sep 25 04:22:33 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 315 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:33.492914 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:33.497648 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b7ad694-2d8e-4845-bd1d-214cc96ad247 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:33.569151 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f8ec1fb-cba0-4843-af73-18bd74681345 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:33.618319 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:33.648178 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0fe3314-d4fd-411b-9678-a6fa4f99a931 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:33.648861 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0fe3314-d4fd-411b-9678-a6fa4f99a931 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:33.649141 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0fe3314-d4fd-411b-9678-a6fa4f99a931 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:33.649409 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0fe3314-d4fd-411b-9678-a6fa4f99a931 None admin] PATH_INFO: `/v3/projects/12400184cf3844998e5e6d8078b8cda4` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:33.651883 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0fe3314-d4fd-411b-9678-a6fa4f99a931 None admin] RBAC: Authorizing `identity:get_project(project_id=12400184cf3844998e5e6d8078b8cda4)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:33.653805 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0fe3314-d4fd-411b-9678-a6fa4f99a931 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:33.655662 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 180/374] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Sep 25 04:22:33 2022] GET /identity/v3/projects/12400184cf3844998e5e6d8078b8cda4 => generated 302 bytes in 275 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:22:33.662219 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0fe3314-d4fd-411b-9678-a6fa4f99a931 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:33.708107 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b7ad694-2d8e-4845-bd1d-214cc96ad247 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:33.775417 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6bd781c-cdd1-4cd0-a4fb-4c5fdc44788c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:33.776077 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6bd781c-cdd1-4cd0-a4fb-4c5fdc44788c None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:33.776363 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6bd781c-cdd1-4cd0-a4fb-4c5fdc44788c None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:33.776656 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6bd781c-cdd1-4cd0-a4fb-4c5fdc44788c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:33.791690 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0fe3314-d4fd-411b-9678-a6fa4f99a931 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:33.839919 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:33.858707 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d4d8809-1bbc-483f-91e0-02841c61bc89 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:33.859348 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d4d8809-1bbc-483f-91e0-02841c61bc89 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:33.859629 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d4d8809-1bbc-483f-91e0-02841c61bc89 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:33.860437 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d4d8809-1bbc-483f-91e0-02841c61bc89 None admin] PATH_INFO: `/v3/groups/1f61c67adfcd46cfadf3d92d2ae29f7a` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:33.862636 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d4d8809-1bbc-483f-91e0-02841c61bc89 None admin] RBAC: Authorizing `identity:get_group(group_id=1f61c67adfcd46cfadf3d92d2ae29f7a)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:33.868134 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d4d8809-1bbc-483f-91e0-02841c61bc89 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:33.872197 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 181/375] 10.222.0.25 () {56 vars in 1180 bytes} [Sun Sep 25 04:22:33 2022] GET /identity/v3/groups/1f61c67adfcd46cfadf3d92d2ae29f7a => generated 226 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:33.876943 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d4d8809-1bbc-483f-91e0-02841c61bc89 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:33.927164 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6bd781c-cdd1-4cd0-a4fb-4c5fdc44788c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:33.928665 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6bd781c-cdd1-4cd0-a4fb-4c5fdc44788c None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:34.035415 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:34.065907 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d4d8809-1bbc-483f-91e0-02841c61bc89 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:34.102217 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 195/376] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Sep 25 04:22:33 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 605 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 25 04:22:34.103980 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f0dcea7-0706-429d-a689-0392b403d25b None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:34.104558 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f0dcea7-0706-429d-a689-0392b403d25b None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:34.105105 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f0dcea7-0706-429d-a689-0392b403d25b None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:34.107133 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 196/377] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:22:34 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:22:34.108052 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f0dcea7-0706-429d-a689-0392b403d25b None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:34.116084 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15ae2928-7029-474c-ab2e-a06cb7964487 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:34.116737 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15ae2928-7029-474c-ab2e-a06cb7964487 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:34.117024 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15ae2928-7029-474c-ab2e-a06cb7964487 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:34.117299 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15ae2928-7029-474c-ab2e-a06cb7964487 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:34.118033 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15ae2928-7029-474c-ab2e-a06cb7964487 None admin] RBAC: Adding query filter params (group.id=1f61c67adfcd46cfadf3d92d2ae29f7a, role.id=02a8d263519b4ff1b511a8577293126c, scope.project.id=12400184cf3844998e5e6d8078b8cda4) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:34.118343 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15ae2928-7029-474c-ab2e-a06cb7964487 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:34.119787 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15ae2928-7029-474c-ab2e-a06cb7964487 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:34.127897 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 182/378] 10.222.0.25 () {56 vars in 1376 bytes} [Sun Sep 25 04:22:33 2022] GET /identity/v3/role_assignments?group.id=1f61c67adfcd46cfadf3d92d2ae29f7a&scope.project.id=12400184cf3844998e5e6d8078b8cda4&role.id=02a8d263519b4ff1b511a8577293126c => generated 260 bytes in 251 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:34.130655 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8dc65a2-8dbc-4875-9715-2bb5a4b4c403 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:34.130655 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8dc65a2-8dbc-4875-9715-2bb5a4b4c403 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:34.130655 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8dc65a2-8dbc-4875-9715-2bb5a4b4c403 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:34.165141 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-c8dc65a2-8dbc-4875-9715-2bb5a4b4c403 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:22:34.167637 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-c8dc65a2-8dbc-4875-9715-2bb5a4b4c403 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:22:34.242772 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:34.267258 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 183/379] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:22:34 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 04:22:34.304756 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f0dcea7-0706-429d-a689-0392b403d25b None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:34.350723 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8552df84-965c-4027-9561-7205e730cdac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:34.351410 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8552df84-965c-4027-9561-7205e730cdac None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:34.351688 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8552df84-965c-4027-9561-7205e730cdac None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:34.351956 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8552df84-965c-4027-9561-7205e730cdac None admin] PATH_INFO: `/v3/projects/99bb568eb0e94976a15231a76508d22f` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:34.353233 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8552df84-965c-4027-9561-7205e730cdac None admin] RBAC: Authorizing `identity:get_project(project_id=99bb568eb0e94976a15231a76508d22f)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:34.354687 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8552df84-965c-4027-9561-7205e730cdac None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:34.356595 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 197/380] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Sep 25 04:22:34 2022] GET /identity/v3/projects/99bb568eb0e94976a15231a76508d22f => generated 344 bytes in 249 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:34.362381 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8dc65a2-8dbc-4875-9715-2bb5a4b4c403 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:34.489262 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8dc65a2-8dbc-4875-9715-2bb5a4b4c403 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:34.538286 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1d4e21e-c06d-48fe-8eaa-1ed1018d37b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:34.538938 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1d4e21e-c06d-48fe-8eaa-1ed1018d37b8 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:34.539222 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1d4e21e-c06d-48fe-8eaa-1ed1018d37b8 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:34.539499 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1d4e21e-c06d-48fe-8eaa-1ed1018d37b8 None admin] PATH_INFO: `/v3/groups/4a4093096641415eba8c947f4f96eabb` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:34.550366 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1d4e21e-c06d-48fe-8eaa-1ed1018d37b8 None admin] RBAC: Authorizing `identity:get_group(group_id=4a4093096641415eba8c947f4f96eabb)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:34.551901 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1d4e21e-c06d-48fe-8eaa-1ed1018d37b8 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:34.553697 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 184/381] 10.222.0.25 () {56 vars in 1180 bytes} [Sun Sep 25 04:22:34 2022] GET /identity/v3/groups/4a4093096641415eba8c947f4f96eabb => generated 229 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:34.559550 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8552df84-965c-4027-9561-7205e730cdac None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:34.611829 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:34.709577 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8552df84-965c-4027-9561-7205e730cdac None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:34.725865 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73a7757e-53d3-4af6-a40c-ac1da26d041f None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:34.726150 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73a7757e-53d3-4af6-a40c-ac1da26d041f None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:34.726430 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73a7757e-53d3-4af6-a40c-ac1da26d041f None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:34.727909 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 185/382] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:22:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:22:34.733608 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f3fc86-d8af-4d32-b9a3-35da49fc7460 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:34.733897 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f3fc86-d8af-4d32-b9a3-35da49fc7460 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:34.734177 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f3fc86-d8af-4d32-b9a3-35da49fc7460 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:34.758327 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-520c6567-c5f4-45a4-aa13-d26ef4a5100f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:34.758958 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-520c6567-c5f4-45a4-aa13-d26ef4a5100f None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:34.759240 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-520c6567-c5f4-45a4-aa13-d26ef4a5100f None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:34.759509 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-520c6567-c5f4-45a4-aa13-d26ef4a5100f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:34.760247 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-520c6567-c5f4-45a4-aa13-d26ef4a5100f None admin] RBAC: Adding query filter params (group.id=4a4093096641415eba8c947f4f96eabb, role.id=c51fe7c5ad804040879acfb54269455f, scope.project.id=99bb568eb0e94976a15231a76508d22f) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:34.760557 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-520c6567-c5f4-45a4-aa13-d26ef4a5100f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:34.762000 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-520c6567-c5f4-45a4-aa13-d26ef4a5100f None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:34.771958 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-01f3fc86-d8af-4d32-b9a3-35da49fc7460 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:22:34.773205 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-01f3fc86-d8af-4d32-b9a3-35da49fc7460 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:22:34.777228 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 198/383] 10.222.0.25 () {56 vars in 1376 bytes} [Sun Sep 25 04:22:34 2022] GET /identity/v3/role_assignments?group.id=4a4093096641415eba8c947f4f96eabb&scope.project.id=99bb568eb0e94976a15231a76508d22f&role.id=c51fe7c5ad804040879acfb54269455f => generated 260 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:34.778097 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-520c6567-c5f4-45a4-aa13-d26ef4a5100f None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:34.859620 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:34.891671 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 186/384] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:22:34 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 159 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 25 04:22:34.947672 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:34.960537 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-520c6567-c5f4-45a4-aa13-d26ef4a5100f None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:35.004618 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79fe8873-2717-4f15-9381-d60f07568228 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:35.005296 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79fe8873-2717-4f15-9381-d60f07568228 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:35.005578 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79fe8873-2717-4f15-9381-d60f07568228 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:35.005858 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79fe8873-2717-4f15-9381-d60f07568228 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:35.017241 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79fe8873-2717-4f15-9381-d60f07568228 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 04:22:35.027073 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-79fe8873-2717-4f15-9381-d60f07568228 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 25 04:22:35.027073 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:22:35.027073 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:22:35.027073 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:22:35.027073 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:22:35.027073 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:22:35.027073 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:22:35.027073 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:22:35.027073 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:22:35.027073 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:22:35.027073 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:22:35.027073 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:22:35.027073 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 04:22:35.027073 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 04:22:35.027073 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 04:22:35.027073 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 04:22:35.028019 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 04:22:35.028019 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 04:22:35.028019 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 04:22:35.028019 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 04:22:35.028019 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:22:35.028019 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:22:35.028019 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:22:35.028019 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:22:35.028019 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:22:35.028019 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:22:35.028019 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:22:35.028019 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:22:35.028019 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:22:35.028019 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:22:35.028019 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:22:35.028019 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:22:35.028893 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:22:35.028893 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:22:35.028893 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:22:35.028893 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:22:35.028893 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 04:22:35.028893 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 04:22:35.028893 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 04:22:35.028893 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 04:22:35.028893 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 04:22:35.028893 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 04:22:35.028893 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 25 04:22:35.028893 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:22:35.028893 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 199/385] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Sep 25 04:22:34 2022] GET /identity/v3/roles/member => generated 84 bytes in 251 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:22:35.036020 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01f3fc86-d8af-4d32-b9a3-35da49fc7460 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:35.177699 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01f3fc86-d8af-4d32-b9a3-35da49fc7460 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:35.215679 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ec9504f-68f4-4753-949f-6398395c74a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:35.216304 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec9504f-68f4-4753-949f-6398395c74a9 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:35.216589 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec9504f-68f4-4753-949f-6398395c74a9 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:35.216895 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec9504f-68f4-4753-949f-6398395c74a9 None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:35.217564 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ec9504f-68f4-4753-949f-6398395c74a9 None admin] RBAC: Adding query filter params (name=member) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:35.217868 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ec9504f-68f4-4753-949f-6398395c74a9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:35.219289 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ec9504f-68f4-4753-949f-6398395c74a9 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:35.228091 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79fe8873-2717-4f15-9381-d60f07568228 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:35.230133 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 187/386] 10.222.0.25 () {56 vars in 1101 bytes} [Sun Sep 25 04:22:35 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:35.238632 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ec9504f-68f4-4753-949f-6398395c74a9 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:35.333011 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:35.403513 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79fe8873-2717-4f15-9381-d60f07568228 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:35.443042 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd2ed27a-60ed-434d-9416-1d55a80d8f7c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:35.443766 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd2ed27a-60ed-434d-9416-1d55a80d8f7c None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:35.444051 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd2ed27a-60ed-434d-9416-1d55a80d8f7c None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:35.444329 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd2ed27a-60ed-434d-9416-1d55a80d8f7c None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:35.454689 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ec9504f-68f4-4753-949f-6398395c74a9 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:35.455703 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd2ed27a-60ed-434d-9416-1d55a80d8f7c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 04:22:35.462286 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-fd2ed27a-60ed-434d-9416-1d55a80d8f7c None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 25 04:22:35.462286 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:22:35.462286 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:22:35.462286 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:22:35.462286 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:22:35.462286 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:22:35.462286 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:22:35.462286 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:22:35.462286 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:22:35.462286 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:22:35.462286 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:22:35.462286 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:22:35.462286 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 04:22:35.462286 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 04:22:35.462286 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 04:22:35.462286 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 04:22:35.463209 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 04:22:35.463209 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 04:22:35.463209 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 04:22:35.463209 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 04:22:35.463209 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:22:35.463209 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:22:35.463209 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:22:35.463209 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:22:35.463209 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:22:35.463209 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:22:35.463209 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:22:35.463209 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:22:35.463209 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:22:35.463209 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:22:35.463209 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:22:35.463209 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:22:35.464005 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:22:35.464005 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:22:35.464005 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:22:35.464005 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:22:35.464005 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 04:22:35.464005 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 04:22:35.464005 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 04:22:35.464005 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 04:22:35.464005 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 04:22:35.464005 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 04:22:35.464005 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 25 04:22:35.464005 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:22:35.464005 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 200/387] 10.222.0.25 () {56 vars in 1114 bytes} [Sun Sep 25 04:22:35 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 236 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:22:35.473759 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd2ed27a-60ed-434d-9416-1d55a80d8f7c None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:35.492741 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:35.511511 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20423145-e965-4a62-b680-a0a1b3a7fbc2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:35.512158 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20423145-e965-4a62-b680-a0a1b3a7fbc2 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:35.512439 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20423145-e965-4a62-b680-a0a1b3a7fbc2 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:35.512992 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20423145-e965-4a62-b680-a0a1b3a7fbc2 None admin] PATH_INFO: `/v3/groups/1f61c67adfcd46cfadf3d92d2ae29f7a` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:35.515072 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20423145-e965-4a62-b680-a0a1b3a7fbc2 None admin] RBAC: Authorizing `identity:get_group(group_id=1f61c67adfcd46cfadf3d92d2ae29f7a)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:35.519906 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20423145-e965-4a62-b680-a0a1b3a7fbc2 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:35.523182 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 188/388] 10.222.0.25 () {56 vars in 1180 bytes} [Sun Sep 25 04:22:35 2022] GET /identity/v3/groups/1f61c67adfcd46cfadf3d92d2ae29f7a => generated 226 bytes in 285 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:22:35.528228 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20423145-e965-4a62-b680-a0a1b3a7fbc2 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:35.661818 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd2ed27a-60ed-434d-9416-1d55a80d8f7c None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:35.728433 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20423145-e965-4a62-b680-a0a1b3a7fbc2 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:35.730587 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a412d2cd-a12b-400b-a4bf-a3e68f5a08dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:35.731229 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a412d2cd-a12b-400b-a4bf-a3e68f5a08dc None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:35.731491 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a412d2cd-a12b-400b-a4bf-a3e68f5a08dc None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:35.731756 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a412d2cd-a12b-400b-a4bf-a3e68f5a08dc None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:35.732423 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a412d2cd-a12b-400b-a4bf-a3e68f5a08dc None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:35.732748 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a412d2cd-a12b-400b-a4bf-a3e68f5a08dc None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:35.734264 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a412d2cd-a12b-400b-a4bf-a3e68f5a08dc None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:35.747684 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:35.751138 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 201/389] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Sep 25 04:22:35 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 280 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:35.756323 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a412d2cd-a12b-400b-a4bf-a3e68f5a08dc None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:35.808039 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d1335e2-0ac9-42e2-b971-aa438cf8ffcc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:35.808958 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1335e2-0ac9-42e2-b971-aa438cf8ffcc None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:35.809385 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1335e2-0ac9-42e2-b971-aa438cf8ffcc None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:35.809829 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1335e2-0ac9-42e2-b971-aa438cf8ffcc None admin] PATH_INFO: `/v3/projects/bceb73400ebb4062be126679d2f16f6d` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:35.811243 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d1335e2-0ac9-42e2-b971-aa438cf8ffcc None admin] RBAC: Authorizing `identity:get_project(project_id=bceb73400ebb4062be126679d2f16f6d)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:35.815014 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d1335e2-0ac9-42e2-b971-aa438cf8ffcc None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:35.816887 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 189/390] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Sep 25 04:22:35 2022] GET /identity/v3/projects/bceb73400ebb4062be126679d2f16f6d => generated 298 bytes in 289 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:35.824140 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d1335e2-0ac9-42e2-b971-aa438cf8ffcc None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:35.880485 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:35.942778 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a412d2cd-a12b-400b-a4bf-a3e68f5a08dc None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:35.994784 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-889d3be6-8dd2-4894-93d0-653fbd0d478c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:35.995398 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-889d3be6-8dd2-4894-93d0-653fbd0d478c None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:35.995626 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-889d3be6-8dd2-4894-93d0-653fbd0d478c None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:35.995884 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-889d3be6-8dd2-4894-93d0-653fbd0d478c None admin] PATH_INFO: `/v3/groups/1f61c67adfcd46cfadf3d92d2ae29f7a` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:35.999944 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-889d3be6-8dd2-4894-93d0-653fbd0d478c None admin] RBAC: Authorizing `identity:get_group(group_id=1f61c67adfcd46cfadf3d92d2ae29f7a)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:36.003820 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-889d3be6-8dd2-4894-93d0-653fbd0d478c None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:36.007206 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 202/391] 10.222.0.25 () {56 vars in 1180 bytes} [Sun Sep 25 04:22:35 2022] GET /identity/v3/groups/1f61c67adfcd46cfadf3d92d2ae29f7a => generated 226 bytes in 250 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:22:36.008404 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b4d0264-0790-4737-b31b-222e1a972dc0 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:36.008708 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b4d0264-0790-4737-b31b-222e1a972dc0 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:36.008989 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b4d0264-0790-4737-b31b-222e1a972dc0 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:36.012130 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 203/392] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:22:36 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:22:36.014167 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b4d0264-0790-4737-b31b-222e1a972dc0 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:36.020435 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d1335e2-0ac9-42e2-b971-aa438cf8ffcc None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:36.060557 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67d225f4-ffa0-47f9-a1a0-c444d25082d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:36.061280 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67d225f4-ffa0-47f9-a1a0-c444d25082d9 None admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:36.061561 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67d225f4-ffa0-47f9-a1a0-c444d25082d9 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:36.061838 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67d225f4-ffa0-47f9-a1a0-c444d25082d9 None admin] PATH_INFO: `/v3/projects/bceb73400ebb4062be126679d2f16f6d/groups/1f61c67adfcd46cfadf3d92d2ae29f7a/roles/8f6cc574056c4278bc9778f14dc4b411` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:36.063865 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67d225f4-ffa0-47f9-a1a0-c444d25082d9 None admin] RBAC: Authorizing `identity:create_grant(project_id=bceb73400ebb4062be126679d2f16f6d, group_id=1f61c67adfcd46cfadf3d92d2ae29f7a, role_id=8f6cc574056c4278bc9778f14dc4b411)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:36.065403 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67d225f4-ffa0-47f9-a1a0-c444d25082d9 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:36.085337 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 190/393] 10.222.0.25 () {58 vars in 1442 bytes} [Sun Sep 25 04:22:35 2022] PUT /identity/v3/projects/bceb73400ebb4062be126679d2f16f6d/groups/1f61c67adfcd46cfadf3d92d2ae29f7a/roles/8f6cc574056c4278bc9778f14dc4b411 => generated 0 bytes in 261 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:22:36.086819 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e793332-4dc2-44e7-a7bb-27d20826671c None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:36.087204 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e793332-4dc2-44e7-a7bb-27d20826671c None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:36.087489 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e793332-4dc2-44e7-a7bb-27d20826671c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:36.116388 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-5e793332-4dc2-44e7-a7bb-27d20826671c None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:22:36.117998 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-5e793332-4dc2-44e7-a7bb-27d20826671c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:22:36.150703 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b4d0264-0790-4737-b31b-222e1a972dc0 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:36.161694 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:36.205948 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 191/394] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:22:36 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 04:22:36.216746 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cac8acb8-78ff-4a84-b16e-228d4d04f70c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:36.217821 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cac8acb8-78ff-4a84-b16e-228d4d04f70c None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:36.218308 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cac8acb8-78ff-4a84-b16e-228d4d04f70c None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:36.218813 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cac8acb8-78ff-4a84-b16e-228d4d04f70c None admin] PATH_INFO: `/v3/projects/bceb73400ebb4062be126679d2f16f6d` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:36.220997 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cac8acb8-78ff-4a84-b16e-228d4d04f70c None admin] RBAC: Authorizing `identity:get_project(project_id=bceb73400ebb4062be126679d2f16f6d)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:36.222776 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cac8acb8-78ff-4a84-b16e-228d4d04f70c None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:36.228770 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 204/395] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Sep 25 04:22:36 2022] GET /identity/v3/projects/bceb73400ebb4062be126679d2f16f6d => generated 298 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:36.233779 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e793332-4dc2-44e7-a7bb-27d20826671c None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:36.351511 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e793332-4dc2-44e7-a7bb-27d20826671c None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:36.387301 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-543ade9e-1a0e-4d44-b296-2b9d17d0cfb5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:36.387980 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-543ade9e-1a0e-4d44-b296-2b9d17d0cfb5 None admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:36.388263 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-543ade9e-1a0e-4d44-b296-2b9d17d0cfb5 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:36.388539 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-543ade9e-1a0e-4d44-b296-2b9d17d0cfb5 None admin] PATH_INFO: `/v3/projects/bceb73400ebb4062be126679d2f16f6d/groups/1f61c67adfcd46cfadf3d92d2ae29f7a/roles/02a8d263519b4ff1b511a8577293126c` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:36.390815 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-543ade9e-1a0e-4d44-b296-2b9d17d0cfb5 None admin] RBAC: Authorizing `identity:create_grant(project_id=bceb73400ebb4062be126679d2f16f6d, group_id=1f61c67adfcd46cfadf3d92d2ae29f7a, role_id=02a8d263519b4ff1b511a8577293126c)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:36.392511 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-543ade9e-1a0e-4d44-b296-2b9d17d0cfb5 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:36.409821 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 192/396] 10.222.0.25 () {58 vars in 1442 bytes} [Sun Sep 25 04:22:36 2022] PUT /identity/v3/projects/bceb73400ebb4062be126679d2f16f6d/groups/1f61c67adfcd46cfadf3d92d2ae29f7a/roles/02a8d263519b4ff1b511a8577293126c => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:22:36.571103 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cac8acb8-78ff-4a84-b16e-228d4d04f70c None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:36.584335 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:36.705200 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cac8acb8-78ff-4a84-b16e-228d4d04f70c None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:36.745638 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4a25ba0-913f-4f06-991c-1a96dfb439db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:36.746522 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4a25ba0-913f-4f06-991c-1a96dfb439db None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:36.746945 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4a25ba0-913f-4f06-991c-1a96dfb439db None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:36.747356 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4a25ba0-913f-4f06-991c-1a96dfb439db None admin] PATH_INFO: `/v3/roles/member` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:36.760510 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4a25ba0-913f-4f06-991c-1a96dfb439db None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 04:22:36.765900 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-c4a25ba0-913f-4f06-991c-1a96dfb439db None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 25 04:22:36.765900 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:22:36.765900 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:22:36.765900 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:22:36.765900 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:22:36.765900 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:22:36.765900 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:22:36.765900 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:22:36.765900 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:22:36.765900 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:22:36.765900 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:22:36.765900 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:22:36.765900 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 04:22:36.765900 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 04:22:36.765900 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 04:22:36.765900 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 04:22:36.767506 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 04:22:36.767506 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 04:22:36.767506 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 04:22:36.767506 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 04:22:36.767506 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:22:36.767506 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:22:36.767506 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:22:36.767506 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:22:36.767506 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:22:36.767506 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:22:36.767506 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:22:36.767506 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:22:36.767506 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:22:36.767506 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:22:36.767506 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:22:36.767506 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:22:36.769028 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:22:36.769028 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:22:36.769028 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:22:36.769028 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:22:36.769028 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 04:22:36.769028 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 04:22:36.769028 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 04:22:36.769028 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 04:22:36.769028 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 04:22:36.769028 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 04:22:36.769028 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 25 04:22:36.769028 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:22:36.769028 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 205/397] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Sep 25 04:22:36 2022] GET /identity/v3/roles/member => generated 84 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:22:36.774301 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-543ade9e-1a0e-4d44-b296-2b9d17d0cfb5 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:36.818047 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:36.915627 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-543ade9e-1a0e-4d44-b296-2b9d17d0cfb5 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:36.941676 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a5dd5f-0bbe-428e-9f26-9b60a5403e68 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:36.941979 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a5dd5f-0bbe-428e-9f26-9b60a5403e68 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:36.942219 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a5dd5f-0bbe-428e-9f26-9b60a5403e68 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:36.943854 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 206/398] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:22:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:22:36.948934 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc8fea6b-ca5f-4d9d-862e-6000a661386e None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:36.949214 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc8fea6b-ca5f-4d9d-862e-6000a661386e None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:36.949485 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc8fea6b-ca5f-4d9d-862e-6000a661386e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:36.962364 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7676f8e-8d75-46ac-85dd-e953ec8b81c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:36.963316 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7676f8e-8d75-46ac-85dd-e953ec8b81c6 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:36.963797 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7676f8e-8d75-46ac-85dd-e953ec8b81c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:36.964294 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7676f8e-8d75-46ac-85dd-e953ec8b81c6 None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:36.965206 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7676f8e-8d75-46ac-85dd-e953ec8b81c6 None admin] RBAC: Adding query filter params (name=member) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:36.965723 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7676f8e-8d75-46ac-85dd-e953ec8b81c6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:36.967373 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7676f8e-8d75-46ac-85dd-e953ec8b81c6 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:36.972323 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-bc8fea6b-ca5f-4d9d-862e-6000a661386e None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:22:36.973844 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-bc8fea6b-ca5f-4d9d-862e-6000a661386e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:22:36.979056 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 193/399] 10.222.0.25 () {56 vars in 1101 bytes} [Sun Sep 25 04:22:36 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:36.984166 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7676f8e-8d75-46ac-85dd-e953ec8b81c6 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:37.080800 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:37.092247 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 207/400] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:22:36 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 25 04:22:37.135148 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:37.147853 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7676f8e-8d75-46ac-85dd-e953ec8b81c6 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:37.191123 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-377eba3b-92de-445a-971c-96789dfdd2d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:37.191761 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-377eba3b-92de-445a-971c-96789dfdd2d4 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:37.192043 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-377eba3b-92de-445a-971c-96789dfdd2d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:37.192311 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-377eba3b-92de-445a-971c-96789dfdd2d4 None admin] PATH_INFO: `/v3/groups/1f61c67adfcd46cfadf3d92d2ae29f7a` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:37.193463 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-377eba3b-92de-445a-971c-96789dfdd2d4 None admin] RBAC: Authorizing `identity:get_group(group_id=1f61c67adfcd46cfadf3d92d2ae29f7a)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:37.194907 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-377eba3b-92de-445a-971c-96789dfdd2d4 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:37.196809 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 194/401] 10.222.0.25 () {56 vars in 1180 bytes} [Sun Sep 25 04:22:36 2022] GET /identity/v3/groups/1f61c67adfcd46cfadf3d92d2ae29f7a => generated 226 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:37.205746 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc8fea6b-ca5f-4d9d-862e-6000a661386e None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:37.362867 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc8fea6b-ca5f-4d9d-862e-6000a661386e None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:37.419634 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fa9820b-e426-41dc-96ff-66fa255fb56e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:37.420677 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fa9820b-e426-41dc-96ff-66fa255fb56e None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:37.421173 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fa9820b-e426-41dc-96ff-66fa255fb56e None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:37.430432 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fa9820b-e426-41dc-96ff-66fa255fb56e None admin] PATH_INFO: `/v3/projects/12400184cf3844998e5e6d8078b8cda4` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:37.432325 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fa9820b-e426-41dc-96ff-66fa255fb56e None admin] RBAC: Authorizing `identity:get_project(project_id=12400184cf3844998e5e6d8078b8cda4)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:37.434072 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fa9820b-e426-41dc-96ff-66fa255fb56e None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:37.436418 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 208/402] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Sep 25 04:22:37 2022] GET /identity/v3/projects/12400184cf3844998e5e6d8078b8cda4 => generated 302 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:37.443271 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-377eba3b-92de-445a-971c-96789dfdd2d4 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:37.481911 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fa9820b-e426-41dc-96ff-66fa255fb56e None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:37.564492 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:37.610711 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fa9820b-e426-41dc-96ff-66fa255fb56e None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:37.654536 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e43d7710-d7e3-449e-984c-702ddf5c82e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:37.655185 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e43d7710-d7e3-449e-984c-702ddf5c82e9 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:37.655462 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e43d7710-d7e3-449e-984c-702ddf5c82e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:37.655734 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e43d7710-d7e3-449e-984c-702ddf5c82e9 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:37.669805 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e43d7710-d7e3-449e-984c-702ddf5c82e9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 04:22:37.676529 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-e43d7710-d7e3-449e-984c-702ddf5c82e9 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 25 04:22:37.676529 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:22:37.676529 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:22:37.676529 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:22:37.676529 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:22:37.676529 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:22:37.676529 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:22:37.676529 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:22:37.676529 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:22:37.676529 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:22:37.676529 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:22:37.676529 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:22:37.676529 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 04:22:37.676529 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 04:22:37.676529 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 04:22:37.676529 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 04:22:37.677523 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 04:22:37.677523 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 04:22:37.677523 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 04:22:37.677523 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 04:22:37.677523 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:22:37.677523 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:22:37.677523 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:22:37.677523 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:22:37.677523 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:22:37.677523 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:22:37.677523 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:22:37.677523 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:22:37.677523 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:22:37.677523 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:22:37.677523 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:22:37.677523 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:22:37.678946 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:22:37.678946 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:22:37.678946 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:22:37.678946 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:22:37.678946 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 04:22:37.678946 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 04:22:37.678946 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 04:22:37.678946 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 04:22:37.678946 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 04:22:37.678946 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 04:22:37.678946 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 25 04:22:37.678946 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:22:37.678946 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 209/403] 10.222.0.25 () {56 vars in 1114 bytes} [Sun Sep 25 04:22:37 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 04:22:37.679667 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-377eba3b-92de-445a-971c-96789dfdd2d4 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:37.683877 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e43d7710-d7e3-449e-984c-702ddf5c82e9 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:37.765778 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d564f87-cb75-4ae2-94ff-e72a9a73a50b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:37.766564 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d564f87-cb75-4ae2-94ff-e72a9a73a50b None admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:37.766844 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d564f87-cb75-4ae2-94ff-e72a9a73a50b None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:37.767141 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d564f87-cb75-4ae2-94ff-e72a9a73a50b None admin] PATH_INFO: `/v3/projects/12400184cf3844998e5e6d8078b8cda4/groups/1f61c67adfcd46cfadf3d92d2ae29f7a/roles/8f6cc574056c4278bc9778f14dc4b411` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:37.770083 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d564f87-cb75-4ae2-94ff-e72a9a73a50b None admin] RBAC: Authorizing `identity:create_grant(project_id=12400184cf3844998e5e6d8078b8cda4, group_id=1f61c67adfcd46cfadf3d92d2ae29f7a, role_id=8f6cc574056c4278bc9778f14dc4b411)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:37.771553 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d564f87-cb75-4ae2-94ff-e72a9a73a50b None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:37.798093 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e43d7710-d7e3-449e-984c-702ddf5c82e9 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:37.803735 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 195/404] 10.222.0.25 () {58 vars in 1442 bytes} [Sun Sep 25 04:22:37 2022] PUT /identity/v3/projects/12400184cf3844998e5e6d8078b8cda4/groups/1f61c67adfcd46cfadf3d92d2ae29f7a/roles/8f6cc574056c4278bc9778f14dc4b411 => generated 0 bytes in 361 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:22:37.835674 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:37.847266 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed5f88a3-568e-4c6f-b9bb-c9fd5a154b72 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:37.847651 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed5f88a3-568e-4c6f-b9bb-c9fd5a154b72 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:37.847911 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed5f88a3-568e-4c6f-b9bb-c9fd5a154b72 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:37.848091 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c27d5068-0fa9-4062-9bd4-6675ffeb5937 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:37.848726 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c27d5068-0fa9-4062-9bd4-6675ffeb5937 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:37.849114 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c27d5068-0fa9-4062-9bd4-6675ffeb5937 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:37.849556 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 196/405] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:22:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:22:37.849713 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c27d5068-0fa9-4062-9bd4-6675ffeb5937 None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:37.850114 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c27d5068-0fa9-4062-9bd4-6675ffeb5937 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:37.850404 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c27d5068-0fa9-4062-9bd4-6675ffeb5937 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:37.851908 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c27d5068-0fa9-4062-9bd4-6675ffeb5937 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:37.854889 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9be9d45b-1fca-4a8c-87a6-73022f1ddddf None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:37.855164 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9be9d45b-1fca-4a8c-87a6-73022f1ddddf None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:37.855435 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9be9d45b-1fca-4a8c-87a6-73022f1ddddf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:37.867900 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 210/406] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Sep 25 04:22:37 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:22:37.873496 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c27d5068-0fa9-4062-9bd4-6675ffeb5937 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:37.877183 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-9be9d45b-1fca-4a8c-87a6-73022f1ddddf None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:22:37.878463 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-9be9d45b-1fca-4a8c-87a6-73022f1ddddf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:22:37.901160 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:38.019084 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 197/407] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:22:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 25 04:22:38.066619 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c27d5068-0fa9-4062-9bd4-6675ffeb5937 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:38.102659 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5390c39-7d5c-4a63-a44a-48b5a9a5ae74 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:38.103302 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5390c39-7d5c-4a63-a44a-48b5a9a5ae74 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:38.103566 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5390c39-7d5c-4a63-a44a-48b5a9a5ae74 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:38.103839 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5390c39-7d5c-4a63-a44a-48b5a9a5ae74 None admin] PATH_INFO: `/v3/groups/1f61c67adfcd46cfadf3d92d2ae29f7a` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:38.105092 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5390c39-7d5c-4a63-a44a-48b5a9a5ae74 None admin] RBAC: Authorizing `identity:get_group(group_id=1f61c67adfcd46cfadf3d92d2ae29f7a)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:38.106523 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5390c39-7d5c-4a63-a44a-48b5a9a5ae74 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:38.108418 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 211/408] 10.222.0.25 () {56 vars in 1180 bytes} [Sun Sep 25 04:22:37 2022] GET /identity/v3/groups/1f61c67adfcd46cfadf3d92d2ae29f7a => generated 226 bytes in 235 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:22:38.114641 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9be9d45b-1fca-4a8c-87a6-73022f1ddddf None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:38.226042 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9be9d45b-1fca-4a8c-87a6-73022f1ddddf None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:38.270230 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98595ac8-c3d8-46e6-983d-f67e110a8416 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:38.270794 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98595ac8-c3d8-46e6-983d-f67e110a8416 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:38.271072 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98595ac8-c3d8-46e6-983d-f67e110a8416 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:38.271341 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98595ac8-c3d8-46e6-983d-f67e110a8416 None admin] PATH_INFO: `/v3/projects/12400184cf3844998e5e6d8078b8cda4` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:38.272458 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98595ac8-c3d8-46e6-983d-f67e110a8416 None admin] RBAC: Authorizing `identity:get_project(project_id=12400184cf3844998e5e6d8078b8cda4)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:38.273927 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98595ac8-c3d8-46e6-983d-f67e110a8416 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:38.275535 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 198/409] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Sep 25 04:22:38 2022] GET /identity/v3/projects/12400184cf3844998e5e6d8078b8cda4 => generated 302 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:38.281359 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5390c39-7d5c-4a63-a44a-48b5a9a5ae74 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:38.364419 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:38.387137 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98595ac8-c3d8-46e6-983d-f67e110a8416 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:38.412898 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:38.438781 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5390c39-7d5c-4a63-a44a-48b5a9a5ae74 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:38.529401 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f335bca6-8d87-4e41-a14a-de4f7eac0ffb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:38.530001 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f335bca6-8d87-4e41-a14a-de4f7eac0ffb None admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:38.530278 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f335bca6-8d87-4e41-a14a-de4f7eac0ffb None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:38.530549 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f335bca6-8d87-4e41-a14a-de4f7eac0ffb None admin] PATH_INFO: `/v3/projects/12400184cf3844998e5e6d8078b8cda4/groups/1f61c67adfcd46cfadf3d92d2ae29f7a/roles/02a8d263519b4ff1b511a8577293126c` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:38.536837 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98595ac8-c3d8-46e6-983d-f67e110a8416 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:38.541438 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f335bca6-8d87-4e41-a14a-de4f7eac0ffb None admin] RBAC: Authorizing `identity:create_grant(project_id=12400184cf3844998e5e6d8078b8cda4, group_id=1f61c67adfcd46cfadf3d92d2ae29f7a, role_id=02a8d263519b4ff1b511a8577293126c)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:38.542900 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f335bca6-8d87-4e41-a14a-de4f7eac0ffb None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:38.574207 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 212/410] 10.222.0.25 () {58 vars in 1442 bytes} [Sun Sep 25 04:22:38 2022] PUT /identity/v3/projects/12400184cf3844998e5e6d8078b8cda4/groups/1f61c67adfcd46cfadf3d92d2ae29f7a/roles/02a8d263519b4ff1b511a8577293126c => generated 0 bytes in 293 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:22:38.586165 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d185604b-ed26-4778-a297-e2865c0029b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:38.587204 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d185604b-ed26-4778-a297-e2865c0029b9 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:38.587769 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d185604b-ed26-4778-a297-e2865c0029b9 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:38.588335 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d185604b-ed26-4778-a297-e2865c0029b9 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:38.601744 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d185604b-ed26-4778-a297-e2865c0029b9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 04:22:38.607332 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-d185604b-ed26-4778-a297-e2865c0029b9 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 04:22:38.607332 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:22:38.607332 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:22:38.607332 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:22:38.607332 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:22:38.607332 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:22:38.607332 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:22:38.607332 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:22:38.607332 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:22:38.607332 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:22:38.607332 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:22:38.607332 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:22:38.607332 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 04:22:38.607332 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 04:22:38.607332 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 04:22:38.607332 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 04:22:38.608304 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 04:22:38.608304 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 04:22:38.608304 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 04:22:38.608304 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 04:22:38.608304 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:22:38.608304 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:22:38.608304 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:22:38.608304 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:22:38.608304 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:22:38.608304 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:22:38.608304 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:22:38.608304 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:22:38.608304 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:22:38.608304 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:22:38.608304 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:22:38.608304 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:22:38.609173 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:22:38.609173 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:22:38.609173 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:22:38.609173 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:22:38.609173 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 04:22:38.609173 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 04:22:38.609173 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 04:22:38.609173 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 04:22:38.609173 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 04:22:38.609173 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 04:22:38.609173 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 04:22:38.609173 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:22:38.611882 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 199/411] 10.222.0.25 () {56 vars in 1096 bytes} [Sun Sep 25 04:22:38 2022] GET /identity/v3/roles/admin => generated 83 bytes in 225 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 04:22:38.617422 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f335bca6-8d87-4e41-a14a-de4f7eac0ffb None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:38.778289 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f335bca6-8d87-4e41-a14a-de4f7eac0ffb None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:38.805059 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:38.831476 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0714b29-0135-41a4-9394-89150e577215 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:38.832485 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0714b29-0135-41a4-9394-89150e577215 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:38.833019 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0714b29-0135-41a4-9394-89150e577215 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:38.833502 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0714b29-0135-41a4-9394-89150e577215 None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:38.834374 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0714b29-0135-41a4-9394-89150e577215 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:38.834892 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0714b29-0135-41a4-9394-89150e577215 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:38.836535 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0714b29-0135-41a4-9394-89150e577215 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:38.848423 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 213/412] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Sep 25 04:22:38 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:22:38.860663 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d185604b-ed26-4778-a297-e2865c0029b9 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:38.978678 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:39.005113 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d185604b-ed26-4778-a297-e2865c0029b9 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:39.040956 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a238ed5-66a2-44f7-86c8-72af7eb2e9d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:39.042110 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a238ed5-66a2-44f7-86c8-72af7eb2e9d9 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:39.042593 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a238ed5-66a2-44f7-86c8-72af7eb2e9d9 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:39.043062 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a238ed5-66a2-44f7-86c8-72af7eb2e9d9 None admin] PATH_INFO: `/v3/groups/4a4093096641415eba8c947f4f96eabb` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:39.044436 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a238ed5-66a2-44f7-86c8-72af7eb2e9d9 None admin] RBAC: Authorizing `identity:get_group(group_id=4a4093096641415eba8c947f4f96eabb)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:39.046205 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a238ed5-66a2-44f7-86c8-72af7eb2e9d9 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:39.048909 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 200/413] 10.222.0.25 () {56 vars in 1180 bytes} [Sun Sep 25 04:22:38 2022] GET /identity/v3/groups/4a4093096641415eba8c947f4f96eabb => generated 229 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:39.054154 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0714b29-0135-41a4-9394-89150e577215 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:39.062145 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3965f0c2-a928-4220-a242-d4d821c38bbc None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:39.062356 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3965f0c2-a928-4220-a242-d4d821c38bbc None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:39.062629 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3965f0c2-a928-4220-a242-d4d821c38bbc None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:39.065339 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 201/414] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:22:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:22:39.069617 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a037ab7-56b0-4b78-89a1-00a77986e36c None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:39.069899 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a037ab7-56b0-4b78-89a1-00a77986e36c None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:39.070171 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a037ab7-56b0-4b78-89a1-00a77986e36c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:39.105710 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-1a037ab7-56b0-4b78-89a1-00a77986e36c None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:22:39.106917 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-1a037ab7-56b0-4b78-89a1-00a77986e36c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:22:39.219783 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 202/415] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:22:39 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 04:22:39.239580 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0714b29-0135-41a4-9394-89150e577215 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:39.277084 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a149a0ee-b16f-4b30-ad7c-1b292e76a7c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:39.277730 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a149a0ee-b16f-4b30-ad7c-1b292e76a7c2 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:39.278007 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a149a0ee-b16f-4b30-ad7c-1b292e76a7c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:39.278278 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a149a0ee-b16f-4b30-ad7c-1b292e76a7c2 None admin] PATH_INFO: `/v3/projects/99bb568eb0e94976a15231a76508d22f` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:39.279530 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a149a0ee-b16f-4b30-ad7c-1b292e76a7c2 None admin] RBAC: Authorizing `identity:get_project(project_id=99bb568eb0e94976a15231a76508d22f)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:39.281002 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a149a0ee-b16f-4b30-ad7c-1b292e76a7c2 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:39.283460 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 214/416] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Sep 25 04:22:39 2022] GET /identity/v3/projects/99bb568eb0e94976a15231a76508d22f => generated 344 bytes in 230 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:39.292154 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a037ab7-56b0-4b78-89a1-00a77986e36c None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:39.378853 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:39.411660 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a037ab7-56b0-4b78-89a1-00a77986e36c None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:39.455827 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95a2ec60-3e52-435a-84b2-5b4797649be2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:39.456501 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a2ec60-3e52-435a-84b2-5b4797649be2 None admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:39.456812 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a2ec60-3e52-435a-84b2-5b4797649be2 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:39.457086 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a2ec60-3e52-435a-84b2-5b4797649be2 None admin] PATH_INFO: `/v3/projects/99bb568eb0e94976a15231a76508d22f/groups/4a4093096641415eba8c947f4f96eabb/roles/c51fe7c5ad804040879acfb54269455f` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:39.459007 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95a2ec60-3e52-435a-84b2-5b4797649be2 None admin] RBAC: Authorizing `identity:create_grant(project_id=99bb568eb0e94976a15231a76508d22f, group_id=4a4093096641415eba8c947f4f96eabb, role_id=c51fe7c5ad804040879acfb54269455f)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:39.460504 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95a2ec60-3e52-435a-84b2-5b4797649be2 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:39.471217 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-361feb55-a93d-4701-af65-96670cdc5ab9 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:39.471505 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-361feb55-a93d-4701-af65-96670cdc5ab9 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:39.471777 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-361feb55-a93d-4701-af65-96670cdc5ab9 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:39.473415 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 215/417] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:22:39 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:22:39.484395 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43b29b66-f82d-4db2-984f-9a3d0d65245c None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:39.485436 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43b29b66-f82d-4db2-984f-9a3d0d65245c None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:39.485952 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43b29b66-f82d-4db2-984f-9a3d0d65245c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:39.486609 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 203/418] 10.222.0.25 () {58 vars in 1442 bytes} [Sun Sep 25 04:22:39 2022] PUT /identity/v3/projects/99bb568eb0e94976a15231a76508d22f/groups/4a4093096641415eba8c947f4f96eabb/roles/c51fe7c5ad804040879acfb54269455f => generated 0 bytes in 195 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:22:39.513923 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-43b29b66-f82d-4db2-984f-9a3d0d65245c None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:22:39.515417 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-43b29b66-f82d-4db2-984f-9a3d0d65245c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:22:39.565878 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95a2ec60-3e52-435a-84b2-5b4797649be2 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:39.583942 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:39.615805 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 216/419] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:22:39 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 04:22:39.706581 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95a2ec60-3e52-435a-84b2-5b4797649be2 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:39.742647 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d20a95be-0e02-42c7-9ccd-f9900b8ebadd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:39.743290 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d20a95be-0e02-42c7-9ccd-f9900b8ebadd None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:39.743570 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d20a95be-0e02-42c7-9ccd-f9900b8ebadd None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:39.743897 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d20a95be-0e02-42c7-9ccd-f9900b8ebadd None admin] PATH_INFO: `/v3/roles/member` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:39.745884 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:39.755431 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d20a95be-0e02-42c7-9ccd-f9900b8ebadd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 04:22:39.760712 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-d20a95be-0e02-42c7-9ccd-f9900b8ebadd None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 25 04:22:39.760712 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:22:39.760712 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:22:39.760712 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:22:39.760712 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:22:39.760712 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:22:39.760712 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:22:39.760712 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:22:39.760712 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:22:39.760712 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:22:39.760712 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:22:39.760712 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:22:39.760712 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 04:22:39.760712 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 04:22:39.760712 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 04:22:39.760712 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 04:22:39.761783 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 04:22:39.761783 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 04:22:39.761783 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 04:22:39.761783 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 04:22:39.761783 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:22:39.761783 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:22:39.761783 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:22:39.761783 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:22:39.761783 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:22:39.761783 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:22:39.761783 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:22:39.761783 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:22:39.761783 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:22:39.761783 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:22:39.761783 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:22:39.761783 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:22:39.762814 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:22:39.762814 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:22:39.762814 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:22:39.762814 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:22:39.762814 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 04:22:39.762814 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 04:22:39.762814 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 04:22:39.762814 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 04:22:39.762814 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 04:22:39.762814 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 04:22:39.762814 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 25 04:22:39.762814 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:22:39.765539 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 204/420] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Sep 25 04:22:39 2022] GET /identity/v3/roles/member => generated 84 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 04:22:39.771469 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43b29b66-f82d-4db2-984f-9a3d0d65245c None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:39.889818 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43b29b66-f82d-4db2-984f-9a3d0d65245c None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:39.929835 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44ea9700-e5d8-476f-b079-9ebff745de12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:39.930454 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44ea9700-e5d8-476f-b079-9ebff745de12 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:39.930732 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44ea9700-e5d8-476f-b079-9ebff745de12 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:39.931002 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44ea9700-e5d8-476f-b079-9ebff745de12 None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:39.931661 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44ea9700-e5d8-476f-b079-9ebff745de12 None admin] RBAC: Adding query filter params (name=member) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:39.931961 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44ea9700-e5d8-476f-b079-9ebff745de12 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:39.933357 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44ea9700-e5d8-476f-b079-9ebff745de12 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:39.947124 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 217/421] 10.222.0.25 () {56 vars in 1101 bytes} [Sun Sep 25 04:22:39 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:22:39.952627 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d20a95be-0e02-42c7-9ccd-f9900b8ebadd None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:39.961750 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44ea9700-e5d8-476f-b079-9ebff745de12 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:40.039247 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:40.100503 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d20a95be-0e02-42c7-9ccd-f9900b8ebadd None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:40.144369 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44ea9700-e5d8-476f-b079-9ebff745de12 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:40.175520 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39020aef-25b8-43ce-8c8a-97a3a2b7e6d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:40.176126 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39020aef-25b8-43ce-8c8a-97a3a2b7e6d7 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:40.176408 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39020aef-25b8-43ce-8c8a-97a3a2b7e6d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:40.176704 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39020aef-25b8-43ce-8c8a-97a3a2b7e6d7 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:40.208537 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eff7dada-6d9a-4453-8c85-5d70b8a8ff45 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:40.209210 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eff7dada-6d9a-4453-8c85-5d70b8a8ff45 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:40.209488 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eff7dada-6d9a-4453-8c85-5d70b8a8ff45 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:40.209757 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eff7dada-6d9a-4453-8c85-5d70b8a8ff45 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:40.237672 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eff7dada-6d9a-4453-8c85-5d70b8a8ff45 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 04:22:40.243052 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-eff7dada-6d9a-4453-8c85-5d70b8a8ff45 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 25 04:22:40.243052 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:22:40.243052 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:22:40.243052 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:22:40.243052 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:22:40.243052 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:22:40.243052 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:22:40.243052 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:22:40.243052 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:22:40.243052 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:22:40.243052 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:22:40.243052 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:22:40.243052 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 04:22:40.243052 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 04:22:40.243052 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 04:22:40.243052 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 04:22:40.244369 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 04:22:40.244369 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 04:22:40.244369 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 04:22:40.244369 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 04:22:40.244369 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:22:40.244369 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:22:40.244369 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:22:40.244369 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:22:40.244369 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:22:40.244369 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:22:40.244369 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:22:40.244369 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:22:40.244369 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:22:40.244369 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:22:40.244369 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:22:40.244369 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:22:40.245220 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:22:40.245220 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:22:40.245220 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:22:40.245220 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:22:40.245220 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 04:22:40.245220 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 04:22:40.245220 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 04:22:40.245220 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 04:22:40.245220 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 04:22:40.245220 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 04:22:40.245220 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 25 04:22:40.245220 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:22:40.248959 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 218/422] 10.222.0.25 () {56 vars in 1114 bytes} [Sun Sep 25 04:22:39 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 288 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:22:40.249811 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eff7dada-6d9a-4453-8c85-5d70b8a8ff45 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:40.263636 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:40.280033 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39020aef-25b8-43ce-8c8a-97a3a2b7e6d7 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:40.282357 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39020aef-25b8-43ce-8c8a-97a3a2b7e6d7 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:40.428717 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:40.438705 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eff7dada-6d9a-4453-8c85-5d70b8a8ff45 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:40.477140 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 205/423] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Sep 25 04:22:39 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 525 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 25 04:22:40.482360 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39020aef-25b8-43ce-8c8a-97a3a2b7e6d7 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:40.498427 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-296a9491-ce7e-406d-b600-c9cdcdf376db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:40.499353 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-296a9491-ce7e-406d-b600-c9cdcdf376db None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:40.499925 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-296a9491-ce7e-406d-b600-c9cdcdf376db None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:40.500766 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-296a9491-ce7e-406d-b600-c9cdcdf376db None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:40.501616 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-296a9491-ce7e-406d-b600-c9cdcdf376db None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:40.502100 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-296a9491-ce7e-406d-b600-c9cdcdf376db None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:40.503637 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-296a9491-ce7e-406d-b600-c9cdcdf376db None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:40.514534 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 219/424] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Sep 25 04:22:40 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 265 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:40.520836 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-296a9491-ce7e-406d-b600-c9cdcdf376db None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:40.669703 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:40.687028 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39020aef-25b8-43ce-8c8a-97a3a2b7e6d7 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:40.704672 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-296a9491-ce7e-406d-b600-c9cdcdf376db None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:40.735258 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce959038-d5c3-439f-b56b-fe302271c6d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:40.735927 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce959038-d5c3-439f-b56b-fe302271c6d6 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:40.736208 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce959038-d5c3-439f-b56b-fe302271c6d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:40.736473 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce959038-d5c3-439f-b56b-fe302271c6d6 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:40.742797 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffc09808-7a4f-40cb-91ca-ce3495a6dcbe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:40.743670 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc09808-7a4f-40cb-91ca-ce3495a6dcbe None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:40.744170 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc09808-7a4f-40cb-91ca-ce3495a6dcbe None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:40.744732 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc09808-7a4f-40cb-91ca-ce3495a6dcbe None admin] PATH_INFO: `/v3/projects/bceb73400ebb4062be126679d2f16f6d` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:40.746101 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffc09808-7a4f-40cb-91ca-ce3495a6dcbe None admin] RBAC: Authorizing `identity:get_project(project_id=bceb73400ebb4062be126679d2f16f6d)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:40.747780 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffc09808-7a4f-40cb-91ca-ce3495a6dcbe None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:40.750011 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 206/425] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Sep 25 04:22:40 2022] GET /identity/v3/projects/bceb73400ebb4062be126679d2f16f6d => generated 298 bytes in 268 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:22:40.752429 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-561a4b17-705c-40d2-b620-6ee346b2045d None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:40.752948 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-561a4b17-705c-40d2-b620-6ee346b2045d None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:40.753425 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-561a4b17-705c-40d2-b620-6ee346b2045d None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:40.754933 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 207/426] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:22:40 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:22:40.755813 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-561a4b17-705c-40d2-b620-6ee346b2045d None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:40.769018 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:40.811730 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce959038-d5c3-439f-b56b-fe302271c6d6 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:40.814297 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce959038-d5c3-439f-b56b-fe302271c6d6 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:40.862187 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-561a4b17-705c-40d2-b620-6ee346b2045d None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:40.899064 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e32a57a4-e0c2-4fd8-9fbc-0398d990c56d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:40.900063 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e32a57a4-e0c2-4fd8-9fbc-0398d990c56d None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:40.900628 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e32a57a4-e0c2-4fd8-9fbc-0398d990c56d None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:40.901124 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e32a57a4-e0c2-4fd8-9fbc-0398d990c56d None admin] PATH_INFO: `/v3/groups/1f61c67adfcd46cfadf3d92d2ae29f7a` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:40.902505 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e32a57a4-e0c2-4fd8-9fbc-0398d990c56d None admin] RBAC: Authorizing `identity:get_group(group_id=1f61c67adfcd46cfadf3d92d2ae29f7a)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:40.904158 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e32a57a4-e0c2-4fd8-9fbc-0398d990c56d None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:40.906355 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 208/427] 10.222.0.25 () {56 vars in 1180 bytes} [Sun Sep 25 04:22:40 2022] GET /identity/v3/groups/1f61c67adfcd46cfadf3d92d2ae29f7a => generated 226 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:40.908527 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a3d4eb0-835d-433d-b865-81d19e04b3be None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:40.909065 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a3d4eb0-835d-433d-b865-81d19e04b3be None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:40.909584 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a3d4eb0-835d-433d-b865-81d19e04b3be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:40.920736 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 220/428] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Sep 25 04:22:40 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 400 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 25 04:22:40.921661 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce959038-d5c3-439f-b56b-fe302271c6d6 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:40.929909 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-7a3d4eb0-835d-433d-b865-81d19e04b3be None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:22:40.931499 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-7a3d4eb0-835d-433d-b865-81d19e04b3be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:22:40.960233 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:40.986968 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:41.009353 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 209/429] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:22:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 04:22:41.010763 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a3d4eb0-835d-433d-b865-81d19e04b3be None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:41.034254 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce959038-d5c3-439f-b56b-fe302271c6d6 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:41.071831 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67b6fc4b-1ff1-44ec-b843-d177b4b9af79 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:41.072433 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67b6fc4b-1ff1-44ec-b843-d177b4b9af79 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:41.072741 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67b6fc4b-1ff1-44ec-b843-d177b4b9af79 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:41.073010 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67b6fc4b-1ff1-44ec-b843-d177b4b9af79 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:41.073932 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67b6fc4b-1ff1-44ec-b843-d177b4b9af79 None admin] RBAC: Adding query filter params (group.id=1f61c67adfcd46cfadf3d92d2ae29f7a, role.id=8f6cc574056c4278bc9778f14dc4b411, scope.project.id=bceb73400ebb4062be126679d2f16f6d) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:41.074137 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67b6fc4b-1ff1-44ec-b843-d177b4b9af79 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:41.075821 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67b6fc4b-1ff1-44ec-b843-d177b4b9af79 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:41.084445 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 221/430] 10.222.0.25 () {56 vars in 1376 bytes} [Sun Sep 25 04:22:40 2022] GET /identity/v3/role_assignments?group.id=1f61c67adfcd46cfadf3d92d2ae29f7a&scope.project.id=bceb73400ebb4062be126679d2f16f6d&role.id=8f6cc574056c4278bc9778f14dc4b411 => generated 611 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:41.124019 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a3d4eb0-835d-433d-b865-81d19e04b3be None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:41.163465 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d2c1d44-864d-4253-b211-344e28611e92 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:41.164343 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d2c1d44-864d-4253-b211-344e28611e92 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:41.164826 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d2c1d44-864d-4253-b211-344e28611e92 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:41.165242 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d2c1d44-864d-4253-b211-344e28611e92 None admin] PATH_INFO: `/v3/projects/bceb73400ebb4062be126679d2f16f6d` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:41.166630 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d2c1d44-864d-4253-b211-344e28611e92 None admin] RBAC: Authorizing `identity:get_project(project_id=bceb73400ebb4062be126679d2f16f6d)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:41.168238 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d2c1d44-864d-4253-b211-344e28611e92 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:41.170095 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 210/431] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Sep 25 04:22:41 2022] GET /identity/v3/projects/bceb73400ebb4062be126679d2f16f6d => generated 298 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:41.176925 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67b6fc4b-1ff1-44ec-b843-d177b4b9af79 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:41.305447 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67b6fc4b-1ff1-44ec-b843-d177b4b9af79 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:41.332901 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d2c1d44-864d-4253-b211-344e28611e92 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:41.345703 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:41.366481 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8018c7ed-7bbe-489e-bb98-f5888c82af47 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:41.367697 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8018c7ed-7bbe-489e-bb98-f5888c82af47 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:41.368455 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8018c7ed-7bbe-489e-bb98-f5888c82af47 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:41.368981 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8018c7ed-7bbe-489e-bb98-f5888c82af47 None admin] PATH_INFO: `/v3/groups/1f61c67adfcd46cfadf3d92d2ae29f7a` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:41.369775 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:41.372143 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8018c7ed-7bbe-489e-bb98-f5888c82af47 None admin] RBAC: Authorizing `identity:get_group(group_id=1f61c67adfcd46cfadf3d92d2ae29f7a)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:41.374463 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8018c7ed-7bbe-489e-bb98-f5888c82af47 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:41.377263 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 222/432] 10.222.0.25 () {56 vars in 1180 bytes} [Sun Sep 25 04:22:41 2022] GET /identity/v3/groups/1f61c67adfcd46cfadf3d92d2ae29f7a => generated 226 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:22:41.383227 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8018c7ed-7bbe-489e-bb98-f5888c82af47 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:41.462310 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d2c1d44-864d-4253-b211-344e28611e92 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:41.510243 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad50e308-4deb-4201-9308-584e73b2243d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:41.511319 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad50e308-4deb-4201-9308-584e73b2243d None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:41.511691 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad50e308-4deb-4201-9308-584e73b2243d None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:41.512060 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad50e308-4deb-4201-9308-584e73b2243d None admin] PATH_INFO: `/v3/roles/member` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:41.524957 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad50e308-4deb-4201-9308-584e73b2243d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 04:22:41.527464 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8018c7ed-7bbe-489e-bb98-f5888c82af47 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:41.529949 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-ad50e308-4deb-4201-9308-584e73b2243d None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 25 04:22:41.529949 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:22:41.529949 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:22:41.529949 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:22:41.529949 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:22:41.529949 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:22:41.529949 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:22:41.529949 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:22:41.529949 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:22:41.529949 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:22:41.529949 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:22:41.529949 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:22:41.529949 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 04:22:41.529949 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 04:22:41.529949 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 04:22:41.529949 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 04:22:41.530883 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 04:22:41.530883 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 04:22:41.530883 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 04:22:41.530883 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 04:22:41.530883 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:22:41.530883 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:22:41.530883 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:22:41.530883 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:22:41.530883 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:22:41.530883 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:22:41.530883 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:22:41.530883 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:22:41.530883 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:22:41.530883 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:22:41.530883 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:22:41.530883 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:22:41.537231 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:22:41.537231 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:22:41.537231 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:22:41.537231 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:22:41.537231 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 04:22:41.537231 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 04:22:41.537231 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 04:22:41.537231 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 04:22:41.537231 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 04:22:41.537231 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 04:22:41.537231 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 25 04:22:41.537231 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:22:41.537231 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 211/433] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Sep 25 04:22:41 2022] GET /identity/v3/roles/member => generated 84 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 04:22:41.537231 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82c72fe6-1eee-4867-9cc6-5a862830dbb1 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:41.538002 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82c72fe6-1eee-4867-9cc6-5a862830dbb1 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:41.538002 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82c72fe6-1eee-4867-9cc6-5a862830dbb1 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:41.538002 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 212/434] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:22:41 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:22:41.541685 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82c72fe6-1eee-4867-9cc6-5a862830dbb1 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:41.569106 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b39d12a5-3a1c-495c-b033-aaf6da6a372f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:41.569956 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b39d12a5-3a1c-495c-b033-aaf6da6a372f None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:41.570237 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b39d12a5-3a1c-495c-b033-aaf6da6a372f None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:41.570609 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b39d12a5-3a1c-495c-b033-aaf6da6a372f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:41.571345 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b39d12a5-3a1c-495c-b033-aaf6da6a372f None admin] RBAC: Adding query filter params (group.id=1f61c67adfcd46cfadf3d92d2ae29f7a, role.id=02a8d263519b4ff1b511a8577293126c, scope.project.id=bceb73400ebb4062be126679d2f16f6d) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:41.571652 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b39d12a5-3a1c-495c-b033-aaf6da6a372f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:41.573168 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b39d12a5-3a1c-495c-b033-aaf6da6a372f None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:41.582508 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 223/435] 10.222.0.25 () {56 vars in 1376 bytes} [Sun Sep 25 04:22:41 2022] GET /identity/v3/role_assignments?group.id=1f61c67adfcd46cfadf3d92d2ae29f7a&scope.project.id=bceb73400ebb4062be126679d2f16f6d&role.id=02a8d263519b4ff1b511a8577293126c => generated 611 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:41.584337 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e66b88b3-9f92-46e9-afa3-a946eaf7047d None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:41.584739 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e66b88b3-9f92-46e9-afa3-a946eaf7047d None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:41.585283 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e66b88b3-9f92-46e9-afa3-a946eaf7047d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:41.607615 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-e66b88b3-9f92-46e9-afa3-a946eaf7047d None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:22:41.609255 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-e66b88b3-9f92-46e9-afa3-a946eaf7047d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:22:41.662574 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:41.679493 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82c72fe6-1eee-4867-9cc6-5a862830dbb1 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:41.688903 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:41.705662 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 224/436] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:22:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 04:22:41.728597 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42c71c37-e935-4cdf-9ea4-a510f9292a38 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:41.729249 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42c71c37-e935-4cdf-9ea4-a510f9292a38 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:41.729538 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42c71c37-e935-4cdf-9ea4-a510f9292a38 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:41.729796 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42c71c37-e935-4cdf-9ea4-a510f9292a38 None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:41.730469 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42c71c37-e935-4cdf-9ea4-a510f9292a38 None admin] RBAC: Adding query filter params (name=member) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:41.730771 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42c71c37-e935-4cdf-9ea4-a510f9292a38 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:41.732166 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42c71c37-e935-4cdf-9ea4-a510f9292a38 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:41.741605 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 213/437] 10.222.0.25 () {56 vars in 1101 bytes} [Sun Sep 25 04:22:41 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:41.747675 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e66b88b3-9f92-46e9-afa3-a946eaf7047d None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:41.863734 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e66b88b3-9f92-46e9-afa3-a946eaf7047d None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:41.904912 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7454d368-aa0e-4a42-a9cb-912b52500624 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:41.905881 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7454d368-aa0e-4a42-a9cb-912b52500624 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:41.906373 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7454d368-aa0e-4a42-a9cb-912b52500624 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:41.907039 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7454d368-aa0e-4a42-a9cb-912b52500624 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:41.967798 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:41.982314 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7454d368-aa0e-4a42-a9cb-912b52500624 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:41.984097 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7454d368-aa0e-4a42-a9cb-912b52500624 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:42.005667 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42c71c37-e935-4cdf-9ea4-a510f9292a38 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:42.108242 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 225/438] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Sep 25 04:22:41 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 361 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 25 04:22:42.118965 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7454d368-aa0e-4a42-a9cb-912b52500624 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:42.151416 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42c71c37-e935-4cdf-9ea4-a510f9292a38 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:42.188541 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b7fc97d-b052-402e-a90e-d2c99bd8a759 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:42.189211 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b7fc97d-b052-402e-a90e-d2c99bd8a759 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:42.189477 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b7fc97d-b052-402e-a90e-d2c99bd8a759 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:42.189746 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b7fc97d-b052-402e-a90e-d2c99bd8a759 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:42.199781 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b7fc97d-b052-402e-a90e-d2c99bd8a759 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 04:22:42.206214 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-2b7fc97d-b052-402e-a90e-d2c99bd8a759 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 25 04:22:42.206214 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:22:42.206214 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:22:42.206214 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:22:42.206214 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:22:42.206214 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:22:42.206214 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:22:42.206214 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:22:42.206214 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:22:42.206214 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:22:42.206214 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:22:42.206214 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:22:42.206214 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 04:22:42.206214 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 04:22:42.206214 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 04:22:42.206214 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 04:22:42.207336 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 04:22:42.207336 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 04:22:42.207336 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 04:22:42.207336 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 04:22:42.207336 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:22:42.207336 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:22:42.207336 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:22:42.207336 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:22:42.207336 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:22:42.207336 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:22:42.207336 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:22:42.207336 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:22:42.207336 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:22:42.207336 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:22:42.207336 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:22:42.207336 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:22:42.208271 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:22:42.208271 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:22:42.208271 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:22:42.208271 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:22:42.208271 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 04:22:42.208271 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 04:22:42.208271 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 04:22:42.208271 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 04:22:42.208271 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 04:22:42.208271 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 04:22:42.208271 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 25 04:22:42.208271 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:22:42.208271 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 214/439] 10.222.0.25 () {56 vars in 1114 bytes} [Sun Sep 25 04:22:42 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 04:22:42.209017 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74ad2c21-f985-423c-a306-5d2625fa6aa3 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:42.209017 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74ad2c21-f985-423c-a306-5d2625fa6aa3 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:42.209017 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74ad2c21-f985-423c-a306-5d2625fa6aa3 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:42.210163 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 215/440] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:22:42 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:22:42.212745 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74ad2c21-f985-423c-a306-5d2625fa6aa3 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:42.216139 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:42.225620 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:42.232743 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7454d368-aa0e-4a42-a9cb-912b52500624 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:42.267037 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c2cbb5f-e0c9-4a44-a31e-b01e636ab00f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:42.267665 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c2cbb5f-e0c9-4a44-a31e-b01e636ab00f None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:42.267945 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c2cbb5f-e0c9-4a44-a31e-b01e636ab00f None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:42.268217 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c2cbb5f-e0c9-4a44-a31e-b01e636ab00f None admin] PATH_INFO: `/v3/projects/12400184cf3844998e5e6d8078b8cda4` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:42.269372 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c2cbb5f-e0c9-4a44-a31e-b01e636ab00f None admin] RBAC: Authorizing `identity:get_project(project_id=12400184cf3844998e5e6d8078b8cda4)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:42.270812 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c2cbb5f-e0c9-4a44-a31e-b01e636ab00f None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:42.272428 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 226/441] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Sep 25 04:22:42 2022] GET /identity/v3/projects/12400184cf3844998e5e6d8078b8cda4 => generated 302 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:42.273558 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-838ff842-74d7-4aac-93c5-1f9bc11aa802 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:42.273842 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-838ff842-74d7-4aac-93c5-1f9bc11aa802 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:42.274118 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-838ff842-74d7-4aac-93c5-1f9bc11aa802 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:42.291525 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-838ff842-74d7-4aac-93c5-1f9bc11aa802 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:22:42.292766 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-838ff842-74d7-4aac-93c5-1f9bc11aa802 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:22:42.320824 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74ad2c21-f985-423c-a306-5d2625fa6aa3 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:42.354320 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-932a56c6-f0b6-4f43-9169-59e18389e28e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:42.354927 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-932a56c6-f0b6-4f43-9169-59e18389e28e None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:42.355210 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-932a56c6-f0b6-4f43-9169-59e18389e28e None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:42.355483 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-932a56c6-f0b6-4f43-9169-59e18389e28e None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:42.356139 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-932a56c6-f0b6-4f43-9169-59e18389e28e None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:42.356444 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-932a56c6-f0b6-4f43-9169-59e18389e28e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:42.357854 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-932a56c6-f0b6-4f43-9169-59e18389e28e None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:42.359045 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 227/442] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:22:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 04:22:42.360417 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-838ff842-74d7-4aac-93c5-1f9bc11aa802 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:42.373125 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 216/443] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Sep 25 04:22:42 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:22:42.378909 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-932a56c6-f0b6-4f43-9169-59e18389e28e None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:42.447192 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:42.457880 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:42.464260 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-838ff842-74d7-4aac-93c5-1f9bc11aa802 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:42.496474 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-932a56c6-f0b6-4f43-9169-59e18389e28e None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:42.504669 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-851974a9-724d-4766-b27f-bf88a515ecef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:42.505321 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-851974a9-724d-4766-b27f-bf88a515ecef None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:42.505601 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-851974a9-724d-4766-b27f-bf88a515ecef None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:42.505868 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-851974a9-724d-4766-b27f-bf88a515ecef None admin] PATH_INFO: `/v3/groups/1f61c67adfcd46cfadf3d92d2ae29f7a` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:42.506983 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-851974a9-724d-4766-b27f-bf88a515ecef None admin] RBAC: Authorizing `identity:get_group(group_id=1f61c67adfcd46cfadf3d92d2ae29f7a)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:42.508406 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-851974a9-724d-4766-b27f-bf88a515ecef None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:42.510004 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 228/444] 10.222.0.25 () {56 vars in 1180 bytes} [Sun Sep 25 04:22:42 2022] GET /identity/v3/groups/1f61c67adfcd46cfadf3d92d2ae29f7a => generated 226 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:42.515675 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-851974a9-724d-4766-b27f-bf88a515ecef None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:42.538178 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-860748a7-ace0-4eb1-b8eb-d764fcdb4f07 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:42.538783 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-860748a7-ace0-4eb1-b8eb-d764fcdb4f07 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:42.539062 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-860748a7-ace0-4eb1-b8eb-d764fcdb4f07 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:42.539330 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-860748a7-ace0-4eb1-b8eb-d764fcdb4f07 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:42.617185 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-860748a7-ace0-4eb1-b8eb-d764fcdb4f07 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:42.618644 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-860748a7-ace0-4eb1-b8eb-d764fcdb4f07 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:42.621766 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-851974a9-724d-4766-b27f-bf88a515ecef None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:42.655183 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10a20b1e-d8ba-4e37-87ee-7ae2e1f97dc1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:42.655795 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a20b1e-d8ba-4e37-87ee-7ae2e1f97dc1 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:42.656076 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a20b1e-d8ba-4e37-87ee-7ae2e1f97dc1 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:42.656353 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a20b1e-d8ba-4e37-87ee-7ae2e1f97dc1 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:42.657691 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10a20b1e-d8ba-4e37-87ee-7ae2e1f97dc1 None admin] RBAC: Adding query filter params (group.id=1f61c67adfcd46cfadf3d92d2ae29f7a, role.id=8f6cc574056c4278bc9778f14dc4b411, scope.project.id=12400184cf3844998e5e6d8078b8cda4) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:42.657989 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10a20b1e-d8ba-4e37-87ee-7ae2e1f97dc1 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:42.659410 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10a20b1e-d8ba-4e37-87ee-7ae2e1f97dc1 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:42.665089 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 229/445] 10.222.0.25 () {56 vars in 1376 bytes} [Sun Sep 25 04:22:42 2022] GET /identity/v3/role_assignments?group.id=1f61c67adfcd46cfadf3d92d2ae29f7a&scope.project.id=12400184cf3844998e5e6d8078b8cda4&role.id=8f6cc574056c4278bc9778f14dc4b411 => generated 611 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:42.665864 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10a20b1e-d8ba-4e37-87ee-7ae2e1f97dc1 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:42.688568 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:42.689310 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:42.730150 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 217/446] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Sep 25 04:22:42 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 352 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 25 04:22:42.736901 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-860748a7-ace0-4eb1-b8eb-d764fcdb4f07 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:42.790905 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10a20b1e-d8ba-4e37-87ee-7ae2e1f97dc1 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:42.831752 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-084609ef-ee13-4563-9564-3b7642394669 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:42.832503 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-084609ef-ee13-4563-9564-3b7642394669 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:42.832825 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-084609ef-ee13-4563-9564-3b7642394669 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:42.833098 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-084609ef-ee13-4563-9564-3b7642394669 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:42.842448 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-860748a7-ace0-4eb1-b8eb-d764fcdb4f07 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:42.844585 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-084609ef-ee13-4563-9564-3b7642394669 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 04:22:42.851376 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-084609ef-ee13-4563-9564-3b7642394669 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 04:22:42.851376 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:22:42.851376 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:22:42.851376 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:22:42.851376 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:22:42.851376 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:22:42.851376 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:22:42.851376 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:22:42.851376 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:22:42.851376 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:22:42.851376 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:22:42.851376 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:22:42.851376 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 04:22:42.851376 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 04:22:42.851376 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 04:22:42.851376 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 04:22:42.852279 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 04:22:42.852279 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 04:22:42.852279 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 04:22:42.852279 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 04:22:42.852279 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:22:42.852279 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:22:42.852279 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:22:42.852279 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:22:42.852279 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:22:42.852279 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:22:42.852279 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:22:42.852279 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:22:42.852279 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:22:42.852279 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:22:42.852279 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:22:42.852279 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:22:42.853149 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:22:42.853149 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:22:42.853149 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:22:42.853149 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:22:42.853149 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 04:22:42.853149 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 04:22:42.853149 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 04:22:42.853149 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 04:22:42.853149 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 04:22:42.853149 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 04:22:42.853149 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 04:22:42.853149 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:22:42.853149 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 230/447] 10.222.0.25 () {56 vars in 1096 bytes} [Sun Sep 25 04:22:42 2022] GET /identity/v3/roles/admin => generated 83 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:22:42.859071 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-084609ef-ee13-4563-9564-3b7642394669 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:42.882792 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b1143cf-af59-4d2b-abcc-ccf8ced576a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:42.883700 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b1143cf-af59-4d2b-abcc-ccf8ced576a7 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:42.883984 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b1143cf-af59-4d2b-abcc-ccf8ced576a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:42.884252 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b1143cf-af59-4d2b-abcc-ccf8ced576a7 None admin] PATH_INFO: `/v3/projects/12400184cf3844998e5e6d8078b8cda4` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:42.885475 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b1143cf-af59-4d2b-abcc-ccf8ced576a7 None admin] RBAC: Authorizing `identity:get_project(project_id=12400184cf3844998e5e6d8078b8cda4)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:42.887196 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b1143cf-af59-4d2b-abcc-ccf8ced576a7 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:42.888924 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 218/448] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Sep 25 04:22:42 2022] GET /identity/v3/projects/12400184cf3844998e5e6d8078b8cda4 => generated 302 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:22:42.894678 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b1143cf-af59-4d2b-abcc-ccf8ced576a7 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:42.943464 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:42.962058 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:42.989392 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-084609ef-ee13-4563-9564-3b7642394669 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:43.017750 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b1143cf-af59-4d2b-abcc-ccf8ced576a7 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:43.032985 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82300b57-b68d-4ae5-9318-0267353ce8e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:43.033658 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82300b57-b68d-4ae5-9318-0267353ce8e6 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:43.033941 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82300b57-b68d-4ae5-9318-0267353ce8e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:43.034212 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82300b57-b68d-4ae5-9318-0267353ce8e6 None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:43.034871 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82300b57-b68d-4ae5-9318-0267353ce8e6 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:43.035175 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82300b57-b68d-4ae5-9318-0267353ce8e6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:43.036547 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82300b57-b68d-4ae5-9318-0267353ce8e6 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:43.047035 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 231/449] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Sep 25 04:22:42 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:22:43.053646 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82300b57-b68d-4ae5-9318-0267353ce8e6 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:43.061210 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-584ab771-ce80-44d4-839f-6b6b42d84937 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:43.061849 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-584ab771-ce80-44d4-839f-6b6b42d84937 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:43.062133 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-584ab771-ce80-44d4-839f-6b6b42d84937 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:43.062407 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-584ab771-ce80-44d4-839f-6b6b42d84937 None admin] PATH_INFO: `/v3/groups/1f61c67adfcd46cfadf3d92d2ae29f7a` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:43.064119 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-584ab771-ce80-44d4-839f-6b6b42d84937 None admin] RBAC: Authorizing `identity:get_group(group_id=1f61c67adfcd46cfadf3d92d2ae29f7a)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:43.065607 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-584ab771-ce80-44d4-839f-6b6b42d84937 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:43.067178 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 219/450] 10.222.0.25 () {56 vars in 1180 bytes} [Sun Sep 25 04:22:42 2022] GET /identity/v3/groups/1f61c67adfcd46cfadf3d92d2ae29f7a => generated 226 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:22:43.073023 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-584ab771-ce80-44d4-839f-6b6b42d84937 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:43.153038 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82300b57-b68d-4ae5-9318-0267353ce8e6 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:43.174493 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-584ab771-ce80-44d4-839f-6b6b42d84937 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:43.183570 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:43.187404 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a80b8ad-524b-449d-b83a-598adc47518f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:43.188573 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a80b8ad-524b-449d-b83a-598adc47518f None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:43.188871 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a80b8ad-524b-449d-b83a-598adc47518f None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:43.189147 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a80b8ad-524b-449d-b83a-598adc47518f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:43.197633 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:43.211881 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbca595d-7e8e-4291-8120-785459a735f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:43.212481 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbca595d-7e8e-4291-8120-785459a735f4 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:43.212793 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbca595d-7e8e-4291-8120-785459a735f4 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:43.213068 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbca595d-7e8e-4291-8120-785459a735f4 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:43.213786 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbca595d-7e8e-4291-8120-785459a735f4 None admin] RBAC: Adding query filter params (group.id=1f61c67adfcd46cfadf3d92d2ae29f7a, role.id=02a8d263519b4ff1b511a8577293126c, scope.project.id=12400184cf3844998e5e6d8078b8cda4) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:43.214095 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbca595d-7e8e-4291-8120-785459a735f4 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:43.215522 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbca595d-7e8e-4291-8120-785459a735f4 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:43.221228 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 220/451] 10.222.0.25 () {56 vars in 1376 bytes} [Sun Sep 25 04:22:43 2022] GET /identity/v3/role_assignments?group.id=1f61c67adfcd46cfadf3d92d2ae29f7a&scope.project.id=12400184cf3844998e5e6d8078b8cda4&role.id=02a8d263519b4ff1b511a8577293126c => generated 611 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:22:43.273576 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a80b8ad-524b-449d-b83a-598adc47518f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:43.275067 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a80b8ad-524b-449d-b83a-598adc47518f None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:43.379109 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 232/452] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Sep 25 04:22:43 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 326 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 25 04:22:43.384864 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbca595d-7e8e-4291-8120-785459a735f4 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:43.490370 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbca595d-7e8e-4291-8120-785459a735f4 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:43.525491 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fca3fb42-ac76-4d3d-867e-41fdb0f18602 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:43.526363 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fca3fb42-ac76-4d3d-867e-41fdb0f18602 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:43.526785 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fca3fb42-ac76-4d3d-867e-41fdb0f18602 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:43.527193 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fca3fb42-ac76-4d3d-867e-41fdb0f18602 None admin] PATH_INFO: `/v3/projects/99bb568eb0e94976a15231a76508d22f` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:43.528458 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fca3fb42-ac76-4d3d-867e-41fdb0f18602 None admin] RBAC: Authorizing `identity:get_project(project_id=99bb568eb0e94976a15231a76508d22f)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:43.529963 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fca3fb42-ac76-4d3d-867e-41fdb0f18602 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:43.531634 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 221/453] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Sep 25 04:22:43 2022] GET /identity/v3/projects/99bb568eb0e94976a15231a76508d22f => generated 344 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:43.537485 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a80b8ad-524b-449d-b83a-598adc47518f None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:43.577969 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:43.637066 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a80b8ad-524b-449d-b83a-598adc47518f None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:43.668812 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-275df5f9-471e-408b-91fe-d5515ca1f77e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:43.669439 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-275df5f9-471e-408b-91fe-d5515ca1f77e None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:43.669719 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-275df5f9-471e-408b-91fe-d5515ca1f77e None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:43.669990 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-275df5f9-471e-408b-91fe-d5515ca1f77e None admin] PATH_INFO: `/v3/groups/4a4093096641415eba8c947f4f96eabb` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:43.671093 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-275df5f9-471e-408b-91fe-d5515ca1f77e None admin] RBAC: Authorizing `identity:get_group(group_id=4a4093096641415eba8c947f4f96eabb)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:43.673108 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-275df5f9-471e-408b-91fe-d5515ca1f77e None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:43.674658 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 233/454] 10.222.0.25 () {56 vars in 1180 bytes} [Sun Sep 25 04:22:43 2022] GET /identity/v3/groups/4a4093096641415eba8c947f4f96eabb => generated 229 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:43.680518 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fca3fb42-ac76-4d3d-867e-41fdb0f18602 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:43.723170 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:43.779770 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fca3fb42-ac76-4d3d-867e-41fdb0f18602 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:43.813042 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f295305c-27e9-430c-a5df-71c39be3f101 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:43.813790 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f295305c-27e9-430c-a5df-71c39be3f101 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:43.814068 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f295305c-27e9-430c-a5df-71c39be3f101 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:43.814338 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f295305c-27e9-430c-a5df-71c39be3f101 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:43.815045 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f295305c-27e9-430c-a5df-71c39be3f101 None admin] RBAC: Adding query filter params (group.id=4a4093096641415eba8c947f4f96eabb, role.id=c51fe7c5ad804040879acfb54269455f, scope.project.id=99bb568eb0e94976a15231a76508d22f) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:43.815349 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f295305c-27e9-430c-a5df-71c39be3f101 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:43.816804 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f295305c-27e9-430c-a5df-71c39be3f101 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:43.822326 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 222/455] 10.222.0.25 () {56 vars in 1376 bytes} [Sun Sep 25 04:22:43 2022] GET /identity/v3/role_assignments?group.id=4a4093096641415eba8c947f4f96eabb&scope.project.id=99bb568eb0e94976a15231a76508d22f&role.id=c51fe7c5ad804040879acfb54269455f => generated 611 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:46.770841 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1575fc58-2852-4736-8531-3d5e592da86a None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:46.771883 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1575fc58-2852-4736-8531-3d5e592da86a None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:46.772361 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1575fc58-2852-4736-8531-3d5e592da86a None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:46.774407 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 234/456] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:22:46 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:22:46.780022 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95361889-46a0-41a1-854c-016909bcbcbc None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:46.780980 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95361889-46a0-41a1-854c-016909bcbcbc None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:46.782047 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95361889-46a0-41a1-854c-016909bcbcbc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:46.805272 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-95361889-46a0-41a1-854c-016909bcbcbc None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:22:46.806784 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-95361889-46a0-41a1-854c-016909bcbcbc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:22:46.826790 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e45cbba8-bb49-4955-9024-54fb6ce048cd None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:46.827409 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e45cbba8-bb49-4955-9024-54fb6ce048cd None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:46.827912 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e45cbba8-bb49-4955-9024-54fb6ce048cd None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:46.829868 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 235/457] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:22:46 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 25 04:22:46.837058 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e22768a0-fca0-48ae-8ab3-917d678cb2d2 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:46.837587 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e22768a0-fca0-48ae-8ab3-917d678cb2d2 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:46.838053 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e22768a0-fca0-48ae-8ab3-917d678cb2d2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:46.857958 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-e22768a0-fca0-48ae-8ab3-917d678cb2d2 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:22:46.859467 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-e22768a0-fca0-48ae-8ab3-917d678cb2d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:22:46.922747 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 223/458] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:22:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 04:22:46.929637 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 236/459] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:22:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 25 04:22:46.955778 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62115dee-0434-41ca-a22b-1d6ec006e32e None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:46.956063 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62115dee-0434-41ca-a22b-1d6ec006e32e None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:46.956335 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62115dee-0434-41ca-a22b-1d6ec006e32e None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:46.957937 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 224/460] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:22:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:22:46.963068 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b2b635f-486a-419d-aa1a-c73e1735c7b7 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:46.963563 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b2b635f-486a-419d-aa1a-c73e1735c7b7 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:46.964031 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b2b635f-486a-419d-aa1a-c73e1735c7b7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:46.990546 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-0b2b635f-486a-419d-aa1a-c73e1735c7b7 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:22:46.992018 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-0b2b635f-486a-419d-aa1a-c73e1735c7b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:22:47.021747 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:47.073879 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 237/461] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:22:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 04:22:47.223732 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62115dee-0434-41ca-a22b-1d6ec006e32e None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:47.231454 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a385f10-1423-4192-9511-b3f36a2de7ab None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:47.231835 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a385f10-1423-4192-9511-b3f36a2de7ab None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:47.232009 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a385f10-1423-4192-9511-b3f36a2de7ab None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:47.233614 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 238/462] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:22:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:22:47.239743 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a292b0b4-c7d0-4e38-a1ef-d1393e6434ec None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:47.240024 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a292b0b4-c7d0-4e38-a1ef-d1393e6434ec None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:47.240298 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a292b0b4-c7d0-4e38-a1ef-d1393e6434ec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:47.268712 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-a292b0b4-c7d0-4e38-a1ef-d1393e6434ec None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:22:47.268903 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:47.270424 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-a292b0b4-c7d0-4e38-a1ef-d1393e6434ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:22:47.348895 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62115dee-0434-41ca-a22b-1d6ec006e32e None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:47.352977 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 239/463] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:22:47 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 25 04:22:47.353981 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a292b0b4-c7d0-4e38-a1ef-d1393e6434ec None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:47.387020 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf9872a7-d155-48bb-b2be-462dcd14240c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:47.387969 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf9872a7-d155-48bb-b2be-462dcd14240c None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:47.388432 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf9872a7-d155-48bb-b2be-462dcd14240c None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:47.388952 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf9872a7-d155-48bb-b2be-462dcd14240c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:47.400141 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf9872a7-d155-48bb-b2be-462dcd14240c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:47.401884 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf9872a7-d155-48bb-b2be-462dcd14240c None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:47.418245 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-cf9872a7-d155-48bb-b2be-462dcd14240c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:22:47.418245 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:22:47.418245 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:22:47.418245 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:22:47.418245 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:22:47.418245 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:22:47.418245 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:22:47.418245 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:22:47.418245 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:22:47.418245 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:22:47.418245 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:22:47.418245 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:22:47.418245 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 04:22:47.418245 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 04:22:47.418245 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 04:22:47.418245 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 04:22:47.419365 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:22:47.419365 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:22:47.419365 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:22:47.419365 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:22:47.419365 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:22:47.419365 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:22:47.419365 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:22:47.419365 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:22:47.419365 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:22:47.419365 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:22:47.419365 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:22:47.419365 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:22:47.419365 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:22:47.419365 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:22:47.419365 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:22:47.419365 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:22:47.419365 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 04:22:47.420238 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 04:22:47.420238 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:22:47.420238 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:22:47.422779 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 225/464] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:22:47 2022] GET /identity/v3/domains/Default => generated 87 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:22:47.428745 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf9872a7-d155-48bb-b2be-462dcd14240c None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:47.473024 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:47.484310 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a292b0b4-c7d0-4e38-a1ef-d1393e6434ec None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:47.533751 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:47.536251 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5b78f6e-eb10-457f-8ca3-e5d7d694f31e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:47.536930 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5b78f6e-eb10-457f-8ca3-e5d7d694f31e None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:47.537228 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5b78f6e-eb10-457f-8ca3-e5d7d694f31e None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:47.537505 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5b78f6e-eb10-457f-8ca3-e5d7d694f31e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:47.548842 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5b78f6e-eb10-457f-8ca3-e5d7d694f31e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:47.550283 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5b78f6e-eb10-457f-8ca3-e5d7d694f31e None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:47.554609 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf9872a7-d155-48bb-b2be-462dcd14240c None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:47.565805 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-a5b78f6e-eb10-457f-8ca3-e5d7d694f31e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:22:47.565805 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:22:47.565805 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:22:47.565805 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:22:47.565805 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:22:47.565805 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:22:47.565805 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:22:47.565805 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:22:47.565805 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:22:47.565805 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:22:47.565805 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:22:47.565805 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:22:47.565805 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 04:22:47.565805 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 04:22:47.565805 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 04:22:47.565805 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 04:22:47.566760 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:22:47.566760 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:22:47.566760 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:22:47.566760 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:22:47.566760 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:22:47.566760 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:22:47.566760 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:22:47.566760 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:22:47.566760 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:22:47.566760 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:22:47.566760 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:22:47.566760 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:22:47.566760 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:22:47.566760 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:22:47.566760 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:22:47.566760 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:22:47.566760 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 04:22:47.567629 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 04:22:47.567629 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:22:47.567629 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:22:47.567629 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 240/465] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:22:47 2022] GET /identity/v3/domains/Default => generated 87 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:22:47.572373 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5b78f6e-eb10-457f-8ca3-e5d7d694f31e None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:47.594824 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb86566b-b91f-455d-b57d-4ad922fd78ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:47.595854 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb86566b-b91f-455d-b57d-4ad922fd78ec None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:47.596390 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb86566b-b91f-455d-b57d-4ad922fd78ec None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:47.596971 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb86566b-b91f-455d-b57d-4ad922fd78ec None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:47.597809 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb86566b-b91f-455d-b57d-4ad922fd78ec None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:47.598311 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb86566b-b91f-455d-b57d-4ad922fd78ec None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:47.599948 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb86566b-b91f-455d-b57d-4ad922fd78ec None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:47.613748 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 226/466] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Sep 25 04:22:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:22:47.619166 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb86566b-b91f-455d-b57d-4ad922fd78ec None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:47.693891 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5b78f6e-eb10-457f-8ca3-e5d7d694f31e None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:47.733057 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb86566b-b91f-455d-b57d-4ad922fd78ec None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:47.735718 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40294ac4-7250-45f0-8235-71e78e34f31b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:47.736433 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40294ac4-7250-45f0-8235-71e78e34f31b None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:47.736754 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40294ac4-7250-45f0-8235-71e78e34f31b None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:47.737030 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40294ac4-7250-45f0-8235-71e78e34f31b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:47.747773 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40294ac4-7250-45f0-8235-71e78e34f31b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:47.749255 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40294ac4-7250-45f0-8235-71e78e34f31b None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:47.758404 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:47.765781 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-40294ac4-7250-45f0-8235-71e78e34f31b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:22:47.765781 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:22:47.765781 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:22:47.765781 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:22:47.765781 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:22:47.765781 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:22:47.765781 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:22:47.765781 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:22:47.765781 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:22:47.765781 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:22:47.765781 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:22:47.765781 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:22:47.765781 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 04:22:47.765781 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 04:22:47.765781 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 04:22:47.765781 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 04:22:47.768515 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:22:47.768515 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:22:47.768515 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:22:47.768515 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:22:47.768515 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:22:47.768515 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:22:47.768515 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:22:47.768515 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:22:47.768515 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:22:47.768515 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:22:47.768515 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:22:47.768515 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:22:47.768515 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:22:47.768515 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:22:47.768515 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:22:47.768515 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:22:47.768515 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 04:22:47.769433 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 04:22:47.769433 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:22:47.769433 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:22:47.769433 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 241/467] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:22:47 2022] GET /identity/v3/domains/Default => generated 87 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:22:47.769433 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40294ac4-7250-45f0-8235-71e78e34f31b None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:47.774907 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b249294-4136-4e2f-9530-355d75379f64 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:47.775529 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b249294-4136-4e2f-9530-355d75379f64 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:47.775806 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b249294-4136-4e2f-9530-355d75379f64 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:47.776076 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b249294-4136-4e2f-9530-355d75379f64 None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:47.776739 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b249294-4136-4e2f-9530-355d75379f64 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:47.777042 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b249294-4136-4e2f-9530-355d75379f64 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:47.778407 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b249294-4136-4e2f-9530-355d75379f64 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:47.793365 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 227/468] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Sep 25 04:22:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:47.793365 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b249294-4136-4e2f-9530-355d75379f64 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:47.801828 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:47.873260 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40294ac4-7250-45f0-8235-71e78e34f31b None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:47.891668 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b249294-4136-4e2f-9530-355d75379f64 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:47.907971 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33ed5c84-268e-478c-9942-501aa999912e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:47.908581 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ed5c84-268e-478c-9942-501aa999912e None admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:47.908888 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ed5c84-268e-478c-9942-501aa999912e None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:47.909162 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ed5c84-268e-478c-9942-501aa999912e None admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:47.909802 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33ed5c84-268e-478c-9942-501aa999912e None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:47.911323 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33ed5c84-268e-478c-9942-501aa999912e None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:47.926998 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b55a7fe5-4945-443f-bdaa-e68f7380093e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:47.927719 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b55a7fe5-4945-443f-bdaa-e68f7380093e None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:47.928008 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b55a7fe5-4945-443f-bdaa-e68f7380093e None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:47.928284 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b55a7fe5-4945-443f-bdaa-e68f7380093e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:47.932606 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 242/469] 10.222.0.25 () {60 vars in 1132 bytes} [Sun Sep 25 04:22:47 2022] POST /identity/v3/users => generated 248 bytes in 164 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:22:47.933441 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33ed5c84-268e-478c-9942-501aa999912e None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:47.947049 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b55a7fe5-4945-443f-bdaa-e68f7380093e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:47.949469 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b55a7fe5-4945-443f-bdaa-e68f7380093e None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:47.968254 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-b55a7fe5-4945-443f-bdaa-e68f7380093e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:22:47.968254 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:22:47.968254 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:22:47.968254 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:22:47.968254 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:22:47.968254 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:22:47.968254 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:22:47.968254 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:22:47.968254 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:22:47.968254 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:22:47.968254 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:22:47.968254 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:22:47.968254 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 04:22:47.968254 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 04:22:47.968254 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 04:22:47.968254 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 04:22:47.970279 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:22:47.970279 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:22:47.970279 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:22:47.970279 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:22:47.970279 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:22:47.970279 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:22:47.970279 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:22:47.970279 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:22:47.970279 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:22:47.970279 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:22:47.970279 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:22:47.970279 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:22:47.970279 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:22:47.970279 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:22:47.970279 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:22:47.970279 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:22:47.970279 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 04:22:47.971147 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 04:22:47.971147 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:22:47.971147 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:22:47.975733 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 228/470] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:22:47 2022] GET /identity/v3/domains/Default => generated 87 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:22:47.976895 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12a541b3-e5e1-40a5-8690-3c9cd9e608d6 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:47.977190 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12a541b3-e5e1-40a5-8690-3c9cd9e608d6 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:47.977465 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12a541b3-e5e1-40a5-8690-3c9cd9e608d6 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:47.978806 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 229/471] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:22:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:22:47.979503 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12a541b3-e5e1-40a5-8690-3c9cd9e608d6 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:48.003663 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:48.046356 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:48.060253 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33ed5c84-268e-478c-9942-501aa999912e None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:48.096136 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12a541b3-e5e1-40a5-8690-3c9cd9e608d6 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:48.101095 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebcc8fdb-ae00-4d52-830e-48269a13c303 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:48.101732 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebcc8fdb-ae00-4d52-830e-48269a13c303 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:48.102008 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebcc8fdb-ae00-4d52-830e-48269a13c303 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:48.102272 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebcc8fdb-ae00-4d52-830e-48269a13c303 None admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:48.102922 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebcc8fdb-ae00-4d52-830e-48269a13c303 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:48.103225 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebcc8fdb-ae00-4d52-830e-48269a13c303 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:48.104667 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebcc8fdb-ae00-4d52-830e-48269a13c303 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:48.117894 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 243/472] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Sep 25 04:22:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:48.118964 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebcc8fdb-ae00-4d52-830e-48269a13c303 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:48.141655 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61d9d10b-321c-488a-a704-4d46e520220f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:48.142581 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61d9d10b-321c-488a-a704-4d46e520220f None admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:48.142581 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61d9d10b-321c-488a-a704-4d46e520220f None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:48.142840 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61d9d10b-321c-488a-a704-4d46e520220f None admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:48.143436 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61d9d10b-321c-488a-a704-4d46e520220f None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:48.144909 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61d9d10b-321c-488a-a704-4d46e520220f None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:48.172866 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 230/473] 10.222.0.25 () {60 vars in 1132 bytes} [Sun Sep 25 04:22:47 2022] POST /identity/v3/users => generated 246 bytes in 192 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:22:48.173469 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75fdac82-5249-4a9d-8ba9-90297b530904 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:48.176658 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75fdac82-5249-4a9d-8ba9-90297b530904 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:48.176658 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75fdac82-5249-4a9d-8ba9-90297b530904 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:48.197927 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-75fdac82-5249-4a9d-8ba9-90297b530904 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:22:48.199325 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-75fdac82-5249-4a9d-8ba9-90297b530904 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:22:48.240775 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebcc8fdb-ae00-4d52-830e-48269a13c303 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:48.263031 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:48.285694 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93857d27-113a-4b8f-8c79-ea63393ba8af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:48.286595 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93857d27-113a-4b8f-8c79-ea63393ba8af None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:48.287048 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93857d27-113a-4b8f-8c79-ea63393ba8af None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:48.287467 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93857d27-113a-4b8f-8c79-ea63393ba8af None admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:48.288218 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 231/474] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:22:48 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 04:22:48.288704 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93857d27-113a-4b8f-8c79-ea63393ba8af None admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:48.289167 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93857d27-113a-4b8f-8c79-ea63393ba8af None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:48.290923 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93857d27-113a-4b8f-8c79-ea63393ba8af None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:48.293188 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75fdac82-5249-4a9d-8ba9-90297b530904 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:48.316846 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 244/475] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Sep 25 04:22:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:48.322553 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93857d27-113a-4b8f-8c79-ea63393ba8af None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:48.379817 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:48.444983 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75fdac82-5249-4a9d-8ba9-90297b530904 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:48.447462 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93857d27-113a-4b8f-8c79-ea63393ba8af None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:48.487174 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81ee5327-5f29-4c3d-afa8-d115a3015b76 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:48.488051 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81ee5327-5f29-4c3d-afa8-d115a3015b76 None admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:48.488606 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81ee5327-5f29-4c3d-afa8-d115a3015b76 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:48.489421 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81ee5327-5f29-4c3d-afa8-d115a3015b76 None admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:48.490195 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81ee5327-5f29-4c3d-afa8-d115a3015b76 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:48.492178 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81ee5327-5f29-4c3d-afa8-d115a3015b76 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:48.496995 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c12d51df-6221-41e5-9bfa-d56b081c619f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:48.497807 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c12d51df-6221-41e5-9bfa-d56b081c619f None admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:48.498237 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c12d51df-6221-41e5-9bfa-d56b081c619f None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:48.498661 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c12d51df-6221-41e5-9bfa-d56b081c619f None admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:48.499464 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c12d51df-6221-41e5-9bfa-d56b081c619f None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:48.502023 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c12d51df-6221-41e5-9bfa-d56b081c619f None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:48.523096 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 245/476] 10.222.0.25 () {60 vars in 1132 bytes} [Sun Sep 25 04:22:48 2022] POST /identity/v3/users => generated 249 bytes in 201 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 04:22:48.531907 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 232/477] 10.222.0.25 () {60 vars in 1132 bytes} [Sun Sep 25 04:22:48 2022] POST /identity/v3/users => generated 248 bytes in 243 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:22:48.612555 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81ee5327-5f29-4c3d-afa8-d115a3015b76 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:48.650105 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:48.751502 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81ee5327-5f29-4c3d-afa8-d115a3015b76 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:48.788464 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d822648-65b6-464f-9219-c28f930c5934 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:48.789385 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d822648-65b6-464f-9219-c28f930c5934 None admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:48.789808 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d822648-65b6-464f-9219-c28f930c5934 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:48.790225 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d822648-65b6-464f-9219-c28f930c5934 None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:48.791039 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d822648-65b6-464f-9219-c28f930c5934 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:48.792541 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d822648-65b6-464f-9219-c28f930c5934 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:48.802108 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.sql.core [None req-7d822648-65b6-464f-9219-c28f930c5934 None admin] Conflict role: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Sep 25 04:22:48.802108 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Sep 25 04:22:48.802108 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [parameters: {'id': 'd0eb2c151153458bad3473ce9bd9ba89', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Sep 25 04:22:48.802108 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81434) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Sep 25 04:22:48.819375 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-7d822648-65b6-464f-9219-c28f930c5934 None admin] Conflict occurred attempting to store role - Duplicate entry found with name anotherrole.: keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Sep 25 04:22:48.819375 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:22:48.819375 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 25 04:22:48.819375 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 25 04:22:48.819375 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 25 04:22:48.819375 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 25 04:22:48.819375 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 25 04:22:48.819375 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application result = self._query(query) Sep 25 04:22:48.819375 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 25 04:22:48.819375 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application conn.query(q) Sep 25 04:22:48.819375 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 25 04:22:48.819375 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 25 04:22:48.819375 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 25 04:22:48.819375 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application result.read() Sep 25 04:22:48.819375 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 25 04:22:48.820477 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 25 04:22:48.820477 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 25 04:22:48.820477 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 25 04:22:48.820477 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 25 04:22:48.820477 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 25 04:22:48.820477 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 25 04:22:48.820477 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 25 04:22:48.820477 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Sep 25 04:22:48.820477 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:22:48.820477 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Sep 25 04:22:48.820477 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:22:48.820477 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:22:48.820477 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Sep 25 04:22:48.820477 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return method(*args, **kwargs) Sep 25 04:22:48.820477 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 33, in create_role Sep 25 04:22:48.820477 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return ref.to_dict() Sep 25 04:22:48.820477 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 25 04:22:48.820477 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application next(self.gen) Sep 25 04:22:48.821695 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Sep 25 04:22:48.821695 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application yield resource Sep 25 04:22:48.821695 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 25 04:22:48.821695 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application next(self.gen) Sep 25 04:22:48.821695 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Sep 25 04:22:48.821695 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application self.session.rollback() Sep 25 04:22:48.821695 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Sep 25 04:22:48.821695 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application self.force_reraise() Sep 25 04:22:48.821695 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Sep 25 04:22:48.821695 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise self.value Sep 25 04:22:48.821695 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Sep 25 04:22:48.821695 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Sep 25 04:22:48.821695 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Sep 25 04:22:48.821695 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application session.commit() Sep 25 04:22:48.821695 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Sep 25 04:22:48.821695 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Sep 25 04:22:48.821695 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Sep 25 04:22:48.822731 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application self._prepare_impl() Sep 25 04:22:48.822731 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Sep 25 04:22:48.822731 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application self.session.flush() Sep 25 04:22:48.822731 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Sep 25 04:22:48.822731 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application self._flush(objects) Sep 25 04:22:48.822731 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Sep 25 04:22:48.822731 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Sep 25 04:22:48.822731 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Sep 25 04:22:48.822731 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application compat.raise_( Sep 25 04:22:48.822731 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 25 04:22:48.822731 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception Sep 25 04:22:48.822731 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Sep 25 04:22:48.822731 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application flush_context.execute() Sep 25 04:22:48.822731 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Sep 25 04:22:48.822731 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rec.execute(self) Sep 25 04:22:48.822731 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Sep 25 04:22:48.822731 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Sep 25 04:22:48.823660 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Sep 25 04:22:48.823660 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application _emit_insert_statements( Sep 25 04:22:48.823660 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Sep 25 04:22:48.823660 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application c = connection._execute_20( Sep 25 04:22:48.823660 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Sep 25 04:22:48.823660 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Sep 25 04:22:48.823660 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Sep 25 04:22:48.823660 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Sep 25 04:22:48.823660 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Sep 25 04:22:48.823660 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ret = self._execute_context( Sep 25 04:22:48.823660 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Sep 25 04:22:48.823660 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Sep 25 04:22:48.823660 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Sep 25 04:22:48.823660 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Sep 25 04:22:48.823660 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 25 04:22:48.823660 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception Sep 25 04:22:48.824486 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 25 04:22:48.824486 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 25 04:22:48.824486 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 25 04:22:48.824486 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 25 04:22:48.824486 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 25 04:22:48.824486 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application result = self._query(query) Sep 25 04:22:48.824486 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 25 04:22:48.824486 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application conn.query(q) Sep 25 04:22:48.824486 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 25 04:22:48.824486 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 25 04:22:48.824486 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 25 04:22:48.824486 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application result.read() Sep 25 04:22:48.824486 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 25 04:22:48.824486 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 25 04:22:48.824486 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 25 04:22:48.824486 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 25 04:22:48.825582 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 25 04:22:48.825582 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 25 04:22:48.825582 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 25 04:22:48.825582 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 25 04:22:48.825582 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Sep 25 04:22:48.825582 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Sep 25 04:22:48.825582 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [parameters: {'id': 'd0eb2c151153458bad3473ce9bd9ba89', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Sep 25 04:22:48.825582 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Sep 25 04:22:48.825582 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:22:48.825582 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 25 04:22:48.825582 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:22:48.825582 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:22:48.825582 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:22:48.825582 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:22:48.825582 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:22:48.825582 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:22:48.826539 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:22:48.826539 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:22:48.826539 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:22:48.826539 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:22:48.826539 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:22:48.826539 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:22:48.826539 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 115, in post Sep 25 04:22:48.826539 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ref = PROVIDERS.role_api.create_role( Sep 25 04:22:48.826539 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:22:48.826539 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:22:48.826539 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1298, in create_role Sep 25 04:22:48.826539 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ret = self.driver.create_role(role_id, role) Sep 25 04:22:48.826539 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Sep 25 04:22:48.826539 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Sep 25 04:22:48.826539 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Sep 25 04:22:48.826539 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:22:48.827449 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 246/478] 10.222.0.25 () {60 vars in 1131 bytes} [Sun Sep 25 04:22:48 2022] POST /identity/v3/roles => generated 144 bytes in 213 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:22:48.831997 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c12d51df-6221-41e5-9bfa-d56b081c619f None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:48.952627 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:48.956363 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c12d51df-6221-41e5-9bfa-d56b081c619f None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:49.009824 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1de4e284-e423-4c70-9e04-9d13f44bc1af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:49.010443 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de4e284-e423-4c70-9e04-9d13f44bc1af None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:49.010731 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de4e284-e423-4c70-9e04-9d13f44bc1af None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:49.011002 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de4e284-e423-4c70-9e04-9d13f44bc1af None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:49.025160 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1de4e284-e423-4c70-9e04-9d13f44bc1af None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 04:22:49.030569 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-1de4e284-e423-4c70-9e04-9d13f44bc1af None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 25 04:22:49.030569 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:22:49.030569 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:22:49.030569 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:22:49.030569 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:22:49.030569 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:22:49.030569 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:22:49.030569 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:22:49.030569 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:22:49.030569 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:22:49.030569 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:22:49.030569 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:22:49.030569 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 04:22:49.030569 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 04:22:49.030569 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 04:22:49.030569 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 04:22:49.032258 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 04:22:49.032258 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 04:22:49.032258 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 04:22:49.032258 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 04:22:49.032258 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:22:49.032258 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:22:49.032258 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:22:49.032258 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:22:49.032258 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:22:49.032258 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:22:49.032258 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:22:49.032258 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:22:49.032258 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:22:49.032258 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:22:49.032258 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:22:49.032258 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:22:49.033431 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:22:49.033431 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:22:49.033431 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:22:49.033431 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:22:49.033431 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 04:22:49.033431 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 04:22:49.033431 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 04:22:49.033431 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 04:22:49.033431 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 04:22:49.033431 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 04:22:49.033431 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 25 04:22:49.033431 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:22:49.033431 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 233/479] 10.222.0.25 () {56 vars in 1114 bytes} [Sun Sep 25 04:22:48 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:22:49.038105 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d822648-65b6-464f-9219-c28f930c5934 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:49.160179 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:49.166305 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d822648-65b6-464f-9219-c28f930c5934 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:49.203080 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2b37755-05f4-400c-9b66-a85ec39350ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:49.204241 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2b37755-05f4-400c-9b66-a85ec39350ae None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:49.204505 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2b37755-05f4-400c-9b66-a85ec39350ae None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:49.204805 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2b37755-05f4-400c-9b66-a85ec39350ae None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:49.205471 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2b37755-05f4-400c-9b66-a85ec39350ae None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:49.205775 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2b37755-05f4-400c-9b66-a85ec39350ae None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:49.207143 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2b37755-05f4-400c-9b66-a85ec39350ae None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:49.215769 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 247/480] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Sep 25 04:22:49 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:50.779005 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58366264-15f5-4810-97d1-f38e6f83989e None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:50.779521 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58366264-15f5-4810-97d1-f38e6f83989e None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:50.779521 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58366264-15f5-4810-97d1-f38e6f83989e None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:50.781104 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 234/481] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:22:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:22:50.787086 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5705491d-86b8-496d-ba47-f001c3974521 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:50.787496 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5705491d-86b8-496d-ba47-f001c3974521 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:50.787683 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5705491d-86b8-496d-ba47-f001c3974521 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:50.806479 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-5705491d-86b8-496d-ba47-f001c3974521 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:22:50.807666 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-5705491d-86b8-496d-ba47-f001c3974521 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:22:50.894381 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 248/482] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:22:50 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 25 04:22:51.057741 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb4417cf-2b87-49aa-a855-b15249bb5c37 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:51.058036 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb4417cf-2b87-49aa-a855-b15249bb5c37 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:51.058217 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb4417cf-2b87-49aa-a855-b15249bb5c37 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:51.059551 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 235/483] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:22:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:22:51.065117 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8f1f5f9-92f6-42b6-a2cb-f1c7e81ab7bb None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:51.065629 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8f1f5f9-92f6-42b6-a2cb-f1c7e81ab7bb None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:51.066100 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8f1f5f9-92f6-42b6-a2cb-f1c7e81ab7bb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:51.089277 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-d8f1f5f9-92f6-42b6-a2cb-f1c7e81ab7bb None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:22:51.090795 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-d8f1f5f9-92f6-42b6-a2cb-f1c7e81ab7bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:22:51.173651 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 249/484] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:22:51 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 04:22:51.272285 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb4417cf-2b87-49aa-a855-b15249bb5c37 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:51.414298 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb4417cf-2b87-49aa-a855-b15249bb5c37 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:51.430594 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec3e2cd3-a24f-4c5e-a404-a8cfe6b6bec6 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:51.430895 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec3e2cd3-a24f-4c5e-a404-a8cfe6b6bec6 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:51.431147 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec3e2cd3-a24f-4c5e-a404-a8cfe6b6bec6 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:51.436428 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 250/485] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:22:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 25 04:22:51.438203 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fea8de1-9ac0-49eb-836d-9b09ed022b20 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:51.438490 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fea8de1-9ac0-49eb-836d-9b09ed022b20 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:51.438940 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fea8de1-9ac0-49eb-836d-9b09ed022b20 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:51.459166 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-0fea8de1-9ac0-49eb-836d-9b09ed022b20 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:22:51.460361 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-0fea8de1-9ac0-49eb-836d-9b09ed022b20 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:22:51.473504 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4908f79c-7e2c-4b63-8ee7-dc3a338a0e63 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:51.474154 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4908f79c-7e2c-4b63-8ee7-dc3a338a0e63 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:51.474438 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4908f79c-7e2c-4b63-8ee7-dc3a338a0e63 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:51.479659 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4908f79c-7e2c-4b63-8ee7-dc3a338a0e63 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:51.486177 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:51.497243 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4908f79c-7e2c-4b63-8ee7-dc3a338a0e63 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 04:22:51.502769 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-4908f79c-7e2c-4b63-8ee7-dc3a338a0e63 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 25 04:22:51.502769 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:22:51.502769 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:22:51.502769 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:22:51.502769 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:22:51.502769 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:22:51.502769 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:22:51.502769 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:22:51.502769 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:22:51.502769 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:22:51.502769 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:22:51.502769 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:22:51.502769 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 04:22:51.502769 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 04:22:51.502769 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 04:22:51.502769 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 04:22:51.503694 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 04:22:51.503694 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 04:22:51.503694 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 04:22:51.503694 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 04:22:51.503694 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:22:51.503694 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:22:51.503694 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:22:51.503694 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:22:51.503694 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:22:51.503694 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:22:51.503694 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:22:51.503694 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:22:51.503694 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:22:51.503694 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:22:51.503694 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:22:51.503694 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:22:51.504511 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:22:51.504511 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:22:51.504511 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:22:51.504511 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:22:51.504511 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 04:22:51.504511 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 04:22:51.504511 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 04:22:51.504511 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 04:22:51.504511 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 04:22:51.504511 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 04:22:51.504511 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 25 04:22:51.504511 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:22:51.505617 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 236/486] 10.222.0.25 () {56 vars in 1102 bytes} [Sun Sep 25 04:22:51 2022] GET /identity/v3/roles/service => generated 85 bytes in 233 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:22:51.512021 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4908f79c-7e2c-4b63-8ee7-dc3a338a0e63 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:51.556560 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:51.582059 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 251/487] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:22:51 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 25 04:22:51.583082 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fea8de1-9ac0-49eb-836d-9b09ed022b20 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:51.706828 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4908f79c-7e2c-4b63-8ee7-dc3a338a0e63 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:51.759447 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fea8de1-9ac0-49eb-836d-9b09ed022b20 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:51.788233 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d506d001-3a5c-4b7a-93e3-a8f858a072e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:51.788922 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d506d001-3a5c-4b7a-93e3-a8f858a072e6 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:51.789146 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d506d001-3a5c-4b7a-93e3-a8f858a072e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:51.789554 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d506d001-3a5c-4b7a-93e3-a8f858a072e6 None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:51.790187 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d506d001-3a5c-4b7a-93e3-a8f858a072e6 None admin] RBAC: Adding query filter params (name=service) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:51.790493 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d506d001-3a5c-4b7a-93e3-a8f858a072e6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:51.792003 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d506d001-3a5c-4b7a-93e3-a8f858a072e6 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:51.799884 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2fd0a37-f145-4f9b-8822-166eeb92de61 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:51.801169 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2fd0a37-f145-4f9b-8822-166eeb92de61 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:51.801454 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 237/488] 10.222.0.25 () {56 vars in 1103 bytes} [Sun Sep 25 04:22:51 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 292 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:22:51.802063 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2fd0a37-f145-4f9b-8822-166eeb92de61 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:51.802566 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2fd0a37-f145-4f9b-8822-166eeb92de61 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:51.802671 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1fa4bf7-52d2-4f6f-af57-eabcc794219a None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:51.802939 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1fa4bf7-52d2-4f6f-af57-eabcc794219a None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:51.803212 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1fa4bf7-52d2-4f6f-af57-eabcc794219a None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:51.808790 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 238/489] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:22:51 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:22:51.809585 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1fa4bf7-52d2-4f6f-af57-eabcc794219a None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:51.815695 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2fd0a37-f145-4f9b-8822-166eeb92de61 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 04:22:51.821216 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-c2fd0a37-f145-4f9b-8822-166eeb92de61 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 25 04:22:51.821216 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:22:51.821216 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:22:51.821216 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:22:51.821216 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:22:51.821216 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:22:51.821216 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:22:51.821216 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:22:51.821216 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:22:51.821216 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:22:51.821216 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:22:51.821216 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:22:51.821216 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 04:22:51.821216 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 04:22:51.821216 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 04:22:51.821216 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 04:22:51.822131 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 04:22:51.822131 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 04:22:51.822131 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 04:22:51.822131 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 04:22:51.822131 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:22:51.822131 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:22:51.822131 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:22:51.822131 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:22:51.822131 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:22:51.822131 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:22:51.822131 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:22:51.822131 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:22:51.822131 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:22:51.822131 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:22:51.822131 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:22:51.822131 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:22:51.822940 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:22:51.822940 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:22:51.822940 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:22:51.822940 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:22:51.822940 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 04:22:51.822940 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 04:22:51.822940 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 04:22:51.822940 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 04:22:51.822940 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 04:22:51.822940 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 04:22:51.822940 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 25 04:22:51.822940 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:22:51.825262 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 252/490] 10.222.0.25 () {56 vars in 1102 bytes} [Sun Sep 25 04:22:51 2022] GET /identity/v3/roles/service => generated 85 bytes in 243 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:22:51.827254 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bc2a28a-b4df-48ac-8d4d-0aa1ad9c0459 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:51.827752 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bc2a28a-b4df-48ac-8d4d-0aa1ad9c0459 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:51.828805 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bc2a28a-b4df-48ac-8d4d-0aa1ad9c0459 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:51.852439 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-3bc2a28a-b4df-48ac-8d4d-0aa1ad9c0459 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:22:51.854097 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-3bc2a28a-b4df-48ac-8d4d-0aa1ad9c0459 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:22:51.856375 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:51.908878 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:51.930026 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1fa4bf7-52d2-4f6f-af57-eabcc794219a None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:51.937619 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 253/491] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:22:51 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 04:22:51.940825 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bc2a28a-b4df-48ac-8d4d-0aa1ad9c0459 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:51.983013 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cccc385b-2e7d-4a01-9caa-bdeebc2372e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:51.985145 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cccc385b-2e7d-4a01-9caa-bdeebc2372e0 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:51.985685 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cccc385b-2e7d-4a01-9caa-bdeebc2372e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:51.986973 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cccc385b-2e7d-4a01-9caa-bdeebc2372e0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:52.002924 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cccc385b-2e7d-4a01-9caa-bdeebc2372e0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:52.004745 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cccc385b-2e7d-4a01-9caa-bdeebc2372e0 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:52.020697 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-cccc385b-2e7d-4a01-9caa-bdeebc2372e0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:22:52.020697 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:22:52.020697 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:22:52.020697 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:22:52.020697 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:22:52.020697 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:22:52.020697 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:22:52.020697 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:22:52.020697 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:22:52.020697 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:22:52.020697 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:22:52.020697 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:22:52.020697 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 04:22:52.020697 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 04:22:52.020697 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 04:22:52.020697 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 04:22:52.021769 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:22:52.021769 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:22:52.021769 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:22:52.021769 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:22:52.021769 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:22:52.021769 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:22:52.021769 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:22:52.021769 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:22:52.021769 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:22:52.021769 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:22:52.021769 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:22:52.021769 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:22:52.021769 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:22:52.021769 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:22:52.021769 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:22:52.021769 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:22:52.021769 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 04:22:52.022644 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 04:22:52.022644 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:22:52.022644 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:22:52.024462 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 239/492] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:22:51 2022] GET /identity/v3/domains/Default => generated 87 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:22:52.025570 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cccc385b-2e7d-4a01-9caa-bdeebc2372e0 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:52.077915 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bc2a28a-b4df-48ac-8d4d-0aa1ad9c0459 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:52.117835 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd9b402e-6620-4a77-b640-3b15aff87485 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:52.118500 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd9b402e-6620-4a77-b640-3b15aff87485 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:52.118724 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd9b402e-6620-4a77-b640-3b15aff87485 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:52.118993 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd9b402e-6620-4a77-b640-3b15aff87485 None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:52.119658 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd9b402e-6620-4a77-b640-3b15aff87485 None admin] RBAC: Adding query filter params (name=service) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:52.119960 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd9b402e-6620-4a77-b640-3b15aff87485 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:52.121356 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd9b402e-6620-4a77-b640-3b15aff87485 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:52.126843 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:52.135494 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 254/493] 10.222.0.25 () {56 vars in 1103 bytes} [Sun Sep 25 04:22:51 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:52.136346 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd9b402e-6620-4a77-b640-3b15aff87485 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:52.137740 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cccc385b-2e7d-4a01-9caa-bdeebc2372e0 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:52.176238 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f6f91bf-c775-414b-ad44-9f2e398a4393 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:52.176899 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f6f91bf-c775-414b-ad44-9f2e398a4393 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:52.177184 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f6f91bf-c775-414b-ad44-9f2e398a4393 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:52.177511 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f6f91bf-c775-414b-ad44-9f2e398a4393 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:52.179381 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:52.189121 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f6f91bf-c775-414b-ad44-9f2e398a4393 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 04:22:52.194410 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-7f6f91bf-c775-414b-ad44-9f2e398a4393 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 25 04:22:52.194410 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:22:52.194410 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:22:52.194410 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:22:52.194410 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:22:52.194410 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:22:52.194410 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:22:52.194410 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:22:52.194410 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:22:52.194410 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:22:52.194410 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:22:52.194410 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:22:52.194410 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 04:22:52.194410 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 04:22:52.194410 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 04:22:52.194410 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 04:22:52.195297 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 04:22:52.195297 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 04:22:52.195297 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 04:22:52.195297 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 04:22:52.195297 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:22:52.195297 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:22:52.195297 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:22:52.195297 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:22:52.195297 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:22:52.195297 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:22:52.195297 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:22:52.195297 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:22:52.195297 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:22:52.195297 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:22:52.195297 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:22:52.195297 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:22:52.196101 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:22:52.196101 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:22:52.196101 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:22:52.196101 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:22:52.196101 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 04:22:52.196101 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 04:22:52.196101 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 04:22:52.196101 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 04:22:52.196101 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 04:22:52.196101 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 04:22:52.196101 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 25 04:22:52.196101 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:22:52.196101 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 240/494] 10.222.0.25 () {56 vars in 1102 bytes} [Sun Sep 25 04:22:52 2022] GET /identity/v3/roles/service => generated 85 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:22:52.196789 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b88f5af6-4bc3-4ea5-aa81-fc1d8dd029d5 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:52.197327 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b88f5af6-4bc3-4ea5-aa81-fc1d8dd029d5 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:52.197327 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b88f5af6-4bc3-4ea5-aa81-fc1d8dd029d5 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:52.202869 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 241/495] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:22:52 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:22:52.202869 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b88f5af6-4bc3-4ea5-aa81-fc1d8dd029d5 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:52.248944 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd9b402e-6620-4a77-b640-3b15aff87485 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:52.281959 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-863b22f0-fda7-4b0f-b90d-4e0c69a4dae2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:52.282574 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-863b22f0-fda7-4b0f-b90d-4e0c69a4dae2 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:52.282852 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-863b22f0-fda7-4b0f-b90d-4e0c69a4dae2 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:52.283122 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-863b22f0-fda7-4b0f-b90d-4e0c69a4dae2 None admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:52.283769 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-863b22f0-fda7-4b0f-b90d-4e0c69a4dae2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:52.284079 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-863b22f0-fda7-4b0f-b90d-4e0c69a4dae2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:52.285511 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-863b22f0-fda7-4b0f-b90d-4e0c69a4dae2 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:52.296560 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 255/496] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Sep 25 04:22:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:52.297464 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-863b22f0-fda7-4b0f-b90d-4e0c69a4dae2 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:52.299255 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b88f5af6-4bc3-4ea5-aa81-fc1d8dd029d5 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:52.332857 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d7cba76-7e86-4c31-82db-7f2bee1e3d7c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:52.333477 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d7cba76-7e86-4c31-82db-7f2bee1e3d7c None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:52.333757 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d7cba76-7e86-4c31-82db-7f2bee1e3d7c None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:52.334029 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d7cba76-7e86-4c31-82db-7f2bee1e3d7c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:52.344262 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d7cba76-7e86-4c31-82db-7f2bee1e3d7c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:52.345762 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d7cba76-7e86-4c31-82db-7f2bee1e3d7c None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:52.358497 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:52.360672 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-8d7cba76-7e86-4c31-82db-7f2bee1e3d7c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:22:52.360672 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:22:52.360672 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:22:52.360672 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:22:52.360672 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:22:52.360672 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:22:52.360672 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:22:52.360672 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:22:52.360672 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:22:52.360672 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:22:52.360672 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:22:52.360672 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:22:52.360672 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 04:22:52.360672 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 04:22:52.360672 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 04:22:52.360672 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 04:22:52.361767 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:22:52.361767 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:22:52.361767 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:22:52.361767 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:22:52.361767 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:22:52.361767 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:22:52.361767 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:22:52.361767 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:22:52.361767 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:22:52.361767 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:22:52.361767 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:22:52.361767 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:22:52.361767 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:22:52.361767 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:22:52.361767 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:22:52.361767 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:22:52.361767 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 04:22:52.362653 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 04:22:52.362653 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:22:52.362653 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:22:52.362653 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 242/497] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:22:52 2022] GET /identity/v3/domains/Default => generated 87 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:22:52.363362 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7151ad6-e22f-46a4-8cbf-03fba094cb20 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:52.363650 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7151ad6-e22f-46a4-8cbf-03fba094cb20 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:52.363922 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7151ad6-e22f-46a4-8cbf-03fba094cb20 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:52.382459 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-a7151ad6-e22f-46a4-8cbf-03fba094cb20 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:22:52.383659 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-a7151ad6-e22f-46a4-8cbf-03fba094cb20 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:22:52.404818 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-863b22f0-fda7-4b0f-b90d-4e0c69a4dae2 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:52.427850 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:52.437828 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1d940ec-d332-4a42-91c5-962ad5d7a600 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:52.438656 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1d940ec-d332-4a42-91c5-962ad5d7a600 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:52.438933 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1d940ec-d332-4a42-91c5-962ad5d7a600 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:52.439210 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1d940ec-d332-4a42-91c5-962ad5d7a600 None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:52.439872 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1d940ec-d332-4a42-91c5-962ad5d7a600 None admin] RBAC: Adding query filter params (name=service) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:52.440172 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1d940ec-d332-4a42-91c5-962ad5d7a600 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:52.441556 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1d940ec-d332-4a42-91c5-962ad5d7a600 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:52.449317 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 256/498] 10.222.0.25 () {56 vars in 1103 bytes} [Sun Sep 25 04:22:52 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:52.450080 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1d940ec-d332-4a42-91c5-962ad5d7a600 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:52.452984 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 243/499] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:22:52 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 04:22:52.454280 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7151ad6-e22f-46a4-8cbf-03fba094cb20 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:52.562213 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7151ad6-e22f-46a4-8cbf-03fba094cb20 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:52.571428 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1d940ec-d332-4a42-91c5-962ad5d7a600 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:52.595468 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:52.597811 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-350a97f8-aa52-4f55-9f84-d086873b31a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:52.598448 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-350a97f8-aa52-4f55-9f84-d086873b31a5 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:52.598730 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-350a97f8-aa52-4f55-9f84-d086873b31a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:52.599005 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-350a97f8-aa52-4f55-9f84-d086873b31a5 None admin] PATH_INFO: `/v3/users/glance` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:52.606418 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-350a97f8-aa52-4f55-9f84-d086873b31a5 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:52.607980 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-350a97f8-aa52-4f55-9f84-d086873b31a5 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:52.609898 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b87bbc1a-46b5-461a-998d-7a3c52e98d2c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:52.610530 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b87bbc1a-46b5-461a-998d-7a3c52e98d2c None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:52.610809 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b87bbc1a-46b5-461a-998d-7a3c52e98d2c None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:52.611492 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b87bbc1a-46b5-461a-998d-7a3c52e98d2c None admin] PATH_INFO: `/v3/roles/service` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:52.621540 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b87bbc1a-46b5-461a-998d-7a3c52e98d2c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 04:22:52.622041 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-350a97f8-aa52-4f55-9f84-d086873b31a5 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Sep 25 04:22:52.622041 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:22:52.622041 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:22:52.622041 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:22:52.622041 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:22:52.622041 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:22:52.622041 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:22:52.622041 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:22:52.622041 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:22:52.622041 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:22:52.622041 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:22:52.622041 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:22:52.622041 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 25 04:22:52.622041 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 25 04:22:52.622041 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 25 04:22:52.622041 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 25 04:22:52.622934 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:22:52.622934 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:22:52.622934 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 25 04:22:52.622934 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 04:22:52.622934 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 25 04:22:52.622934 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 04:22:52.622934 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:22:52.622934 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:22:52.622934 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:22:52.622934 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:22:52.622934 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:22:52.622934 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:22:52.622934 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:22:52.622934 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:22:52.622934 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:22:52.622934 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:22:52.622934 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:22:52.622934 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:22:52.623881 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:22:52.623881 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:22:52.623881 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 25 04:22:52.623881 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 25 04:22:52.623881 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 25 04:22:52.623881 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 25 04:22:52.623881 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 25 04:22:52.623881 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 25 04:22:52.623881 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Sep 25 04:22:52.623881 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:22:52.623881 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 244/500] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Sep 25 04:22:52 2022] GET /identity/v3/users/glance => generated 84 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:22:52.623881 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-350a97f8-aa52-4f55-9f84-d086873b31a5 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:52.632840 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-b87bbc1a-46b5-461a-998d-7a3c52e98d2c None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 25 04:22:52.632840 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:22:52.632840 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:22:52.632840 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:22:52.632840 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:22:52.632840 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:22:52.632840 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:22:52.632840 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:22:52.632840 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:22:52.632840 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:22:52.632840 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:22:52.632840 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:22:52.632840 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 04:22:52.632840 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 04:22:52.632840 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 04:22:52.632840 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 04:22:52.633841 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 04:22:52.633841 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 04:22:52.633841 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 04:22:52.633841 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 04:22:52.633841 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:22:52.633841 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:22:52.633841 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:22:52.633841 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:22:52.633841 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:22:52.633841 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:22:52.633841 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:22:52.633841 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:22:52.633841 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:22:52.633841 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:22:52.633841 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:22:52.633841 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:22:52.634653 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:22:52.634653 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:22:52.634653 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:22:52.634653 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:22:52.634653 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 04:22:52.634653 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 04:22:52.634653 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 04:22:52.634653 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 04:22:52.634653 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 04:22:52.634653 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 04:22:52.634653 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 25 04:22:52.634653 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:22:52.634653 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 257/501] 10.222.0.25 () {56 vars in 1102 bytes} [Sun Sep 25 04:22:52 2022] GET /identity/v3/roles/service => generated 85 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:22:52.635328 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b87bbc1a-46b5-461a-998d-7a3c52e98d2c None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:52.671492 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:52.741891 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b87bbc1a-46b5-461a-998d-7a3c52e98d2c None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:52.747257 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-350a97f8-aa52-4f55-9f84-d086873b31a5 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:52.774664 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1033c669-2f79-49ed-8331-a880304d4de8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:52.775299 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1033c669-2f79-49ed-8331-a880304d4de8 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:52.775588 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1033c669-2f79-49ed-8331-a880304d4de8 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:52.775844 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1033c669-2f79-49ed-8331-a880304d4de8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:52.780797 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84443dd3-73ce-44fc-a24d-a9b2a2e5c347 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:52.782039 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84443dd3-73ce-44fc-a24d-a9b2a2e5c347 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:52.782324 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84443dd3-73ce-44fc-a24d-a9b2a2e5c347 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:52.782600 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84443dd3-73ce-44fc-a24d-a9b2a2e5c347 None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:52.783282 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84443dd3-73ce-44fc-a24d-a9b2a2e5c347 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:52.783588 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84443dd3-73ce-44fc-a24d-a9b2a2e5c347 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:52.784999 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84443dd3-73ce-44fc-a24d-a9b2a2e5c347 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:52.787509 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1033c669-2f79-49ed-8331-a880304d4de8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:52.788973 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1033c669-2f79-49ed-8331-a880304d4de8 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:52.796648 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 245/502] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Sep 25 04:22:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:52.798675 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84443dd3-73ce-44fc-a24d-a9b2a2e5c347 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:52.807560 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-1033c669-2f79-49ed-8331-a880304d4de8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:22:52.807560 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:22:52.807560 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:22:52.807560 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:22:52.807560 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:22:52.807560 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:22:52.807560 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:22:52.807560 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:22:52.807560 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:22:52.807560 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:22:52.807560 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:22:52.807560 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:22:52.807560 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 04:22:52.807560 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 04:22:52.807560 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 04:22:52.807560 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 04:22:52.808945 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:22:52.808945 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:22:52.808945 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:22:52.808945 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:22:52.808945 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:22:52.808945 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:22:52.808945 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:22:52.808945 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:22:52.808945 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:22:52.808945 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:22:52.808945 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:22:52.808945 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:22:52.808945 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:22:52.808945 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:22:52.808945 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:22:52.808945 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:22:52.808945 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 04:22:52.809830 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 04:22:52.809830 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:22:52.809830 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:22:52.809830 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 258/503] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:22:52 2022] GET /identity/v3/domains/Default => generated 87 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:22:52.809830 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1033c669-2f79-49ed-8331-a880304d4de8 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:52.844721 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:52.894753 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:52.900799 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84443dd3-73ce-44fc-a24d-a9b2a2e5c347 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:52.910796 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1033c669-2f79-49ed-8331-a880304d4de8 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:52.934127 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ddb585e-0851-43e8-acaa-bef57e0339ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:52.934734 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ddb585e-0851-43e8-acaa-bef57e0339ad None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:52.935017 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ddb585e-0851-43e8-acaa-bef57e0339ad None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:52.935291 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ddb585e-0851-43e8-acaa-bef57e0339ad None admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:52.936008 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ddb585e-0851-43e8-acaa-bef57e0339ad None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:52.936320 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ddb585e-0851-43e8-acaa-bef57e0339ad None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:52.938370 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ddb585e-0851-43e8-acaa-bef57e0339ad None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:52.944189 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f735279-aa0a-40f5-a8a5-ee4de45d76be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:52.944819 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f735279-aa0a-40f5-a8a5-ee4de45d76be None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:52.945096 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f735279-aa0a-40f5-a8a5-ee4de45d76be None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:52.945361 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f735279-aa0a-40f5-a8a5-ee4de45d76be None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:52.946009 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f735279-aa0a-40f5-a8a5-ee4de45d76be None admin] RBAC: Adding query filter params (name=service) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:52.946309 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f735279-aa0a-40f5-a8a5-ee4de45d76be None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:52.947661 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f735279-aa0a-40f5-a8a5-ee4de45d76be None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:52.957155 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 259/504] 10.222.0.25 () {56 vars in 1103 bytes} [Sun Sep 25 04:22:52 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:52.957935 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f735279-aa0a-40f5-a8a5-ee4de45d76be None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:52.975163 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 246/505] 10.222.0.25 () {56 vars in 1137 bytes} [Sun Sep 25 04:22:52 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:52.976041 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ddb585e-0851-43e8-acaa-bef57e0339ad None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:53.051451 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:53.057829 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f735279-aa0a-40f5-a8a5-ee4de45d76be None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:53.076955 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ddb585e-0851-43e8-acaa-bef57e0339ad None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:53.091837 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aaebf66-9a21-46cc-abbb-83f2fa90df84 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:53.092846 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aaebf66-9a21-46cc-abbb-83f2fa90df84 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:53.093509 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aaebf66-9a21-46cc-abbb-83f2fa90df84 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:53.093845 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aaebf66-9a21-46cc-abbb-83f2fa90df84 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:53.104495 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aaebf66-9a21-46cc-abbb-83f2fa90df84 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:53.105920 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aaebf66-9a21-46cc-abbb-83f2fa90df84 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:53.114013 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb756174-4d7d-446a-aed1-04a05db73baf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:53.114642 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb756174-4d7d-446a-aed1-04a05db73baf None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:53.114919 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb756174-4d7d-446a-aed1-04a05db73baf None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:53.115188 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb756174-4d7d-446a-aed1-04a05db73baf None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:53.120571 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-2aaebf66-9a21-46cc-abbb-83f2fa90df84 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:22:53.120571 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:22:53.120571 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:22:53.120571 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:22:53.120571 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:22:53.120571 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:22:53.120571 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:22:53.120571 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:22:53.120571 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:22:53.120571 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:22:53.120571 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:22:53.120571 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:22:53.120571 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 04:22:53.120571 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 04:22:53.120571 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 04:22:53.120571 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 04:22:53.121684 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:22:53.121684 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:22:53.121684 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:22:53.121684 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:22:53.121684 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:22:53.121684 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:22:53.121684 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:22:53.121684 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:22:53.121684 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:22:53.121684 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:22:53.121684 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:22:53.121684 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:22:53.121684 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:22:53.121684 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:22:53.121684 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:22:53.121684 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:22:53.121684 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 04:22:53.122771 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 04:22:53.122771 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:22:53.122771 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:22:53.122771 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 260/506] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:22:52 2022] GET /identity/v3/domains/Default => generated 87 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:22:53.123065 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aaebf66-9a21-46cc-abbb-83f2fa90df84 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:53.123147 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb756174-4d7d-446a-aed1-04a05db73baf None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:53.124578 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb756174-4d7d-446a-aed1-04a05db73baf None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:53.137282 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-bb756174-4d7d-446a-aed1-04a05db73baf None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Sep 25 04:22:53.137282 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:22:53.137282 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:22:53.137282 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:22:53.137282 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:22:53.137282 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:22:53.137282 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:22:53.137282 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:22:53.137282 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:22:53.137282 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:22:53.137282 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:22:53.137282 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:22:53.137282 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 25 04:22:53.137282 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 25 04:22:53.137282 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 25 04:22:53.137282 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 25 04:22:53.138363 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:22:53.138363 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:22:53.138363 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 25 04:22:53.138363 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 04:22:53.138363 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 25 04:22:53.138363 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 04:22:53.138363 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:22:53.138363 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:22:53.138363 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:22:53.138363 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:22:53.138363 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:22:53.138363 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:22:53.138363 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:22:53.138363 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:22:53.138363 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:22:53.138363 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:22:53.138363 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:22:53.138363 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:22:53.144801 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:22:53.144801 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:22:53.144801 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 25 04:22:53.144801 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 25 04:22:53.144801 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 25 04:22:53.144801 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 25 04:22:53.144801 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 25 04:22:53.144801 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 25 04:22:53.144801 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Sep 25 04:22:53.144801 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:22:53.144801 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 247/507] 10.222.0.25 () {56 vars in 1102 bytes} [Sun Sep 25 04:22:52 2022] GET /identity/v3/users/neutron => generated 85 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:22:53.144801 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb756174-4d7d-446a-aed1-04a05db73baf None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:53.155593 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:53.230742 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aaebf66-9a21-46cc-abbb-83f2fa90df84 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:53.239870 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb756174-4d7d-446a-aed1-04a05db73baf None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:53.263951 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9300be1b-dcd3-4efc-ad1b-8ef638fd33d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:53.264647 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9300be1b-dcd3-4efc-ad1b-8ef638fd33d4 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:53.264956 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9300be1b-dcd3-4efc-ad1b-8ef638fd33d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:53.265234 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9300be1b-dcd3-4efc-ad1b-8ef638fd33d4 None admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:53.265945 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9300be1b-dcd3-4efc-ad1b-8ef638fd33d4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:53.266246 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9300be1b-dcd3-4efc-ad1b-8ef638fd33d4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:53.267607 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9300be1b-dcd3-4efc-ad1b-8ef638fd33d4 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:53.272241 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ce6a13f-f263-4b99-a98a-6aa1adf45691 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:53.272944 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ce6a13f-f263-4b99-a98a-6aa1adf45691 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:53.273224 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ce6a13f-f263-4b99-a98a-6aa1adf45691 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:53.273521 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ce6a13f-f263-4b99-a98a-6aa1adf45691 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:53.279138 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 261/508] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Sep 25 04:22:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:53.279924 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9300be1b-dcd3-4efc-ad1b-8ef638fd33d4 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:53.285425 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ce6a13f-f263-4b99-a98a-6aa1adf45691 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:53.286967 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ce6a13f-f263-4b99-a98a-6aa1adf45691 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:53.290034 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:53.301977 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-0ce6a13f-f263-4b99-a98a-6aa1adf45691 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:22:53.301977 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:22:53.301977 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:22:53.301977 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:22:53.301977 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:22:53.301977 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:22:53.301977 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:22:53.301977 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:22:53.301977 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:22:53.301977 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:22:53.301977 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:22:53.301977 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:22:53.301977 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 04:22:53.301977 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 04:22:53.301977 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 04:22:53.301977 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 04:22:53.303035 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:22:53.303035 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:22:53.303035 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:22:53.303035 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:22:53.303035 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:22:53.303035 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:22:53.303035 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:22:53.303035 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:22:53.303035 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:22:53.303035 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:22:53.303035 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:22:53.303035 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:22:53.303035 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:22:53.303035 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:22:53.303035 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:22:53.303035 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:22:53.303035 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 04:22:53.303895 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 04:22:53.303895 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:22:53.303895 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:22:53.303895 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 248/509] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:22:53 2022] GET /identity/v3/domains/Default => generated 87 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:22:53.303895 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ce6a13f-f263-4b99-a98a-6aa1adf45691 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:53.369528 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:53.385152 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9300be1b-dcd3-4efc-ad1b-8ef638fd33d4 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:53.404275 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ce6a13f-f263-4b99-a98a-6aa1adf45691 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:53.418074 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27232a1e-070f-403d-862e-9b836de83457 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:53.418705 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27232a1e-070f-403d-862e-9b836de83457 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:53.418990 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27232a1e-070f-403d-862e-9b836de83457 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:53.419265 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27232a1e-070f-403d-862e-9b836de83457 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:53.437600 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaee8dcd-3930-409d-900c-b8e8566d55d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:53.438449 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaee8dcd-3930-409d-900c-b8e8566d55d5 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:53.438730 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaee8dcd-3930-409d-900c-b8e8566d55d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:53.439006 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaee8dcd-3930-409d-900c-b8e8566d55d5 None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:53.439640 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaee8dcd-3930-409d-900c-b8e8566d55d5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:53.439947 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaee8dcd-3930-409d-900c-b8e8566d55d5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:53.441353 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaee8dcd-3930-409d-900c-b8e8566d55d5 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:53.453241 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 249/510] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Sep 25 04:22:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:53.454026 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaee8dcd-3930-409d-900c-b8e8566d55d5 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:53.489366 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27232a1e-070f-403d-862e-9b836de83457 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:53.490901 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27232a1e-070f-403d-862e-9b836de83457 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:53.501593 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:53.554694 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaee8dcd-3930-409d-900c-b8e8566d55d5 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:53.580920 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:53.589962 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1881928b-a0bc-4b66-a55b-6540dc919704 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:53.590472 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 262/511] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Sep 25 04:22:53 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 25 04:22:53.590675 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1881928b-a0bc-4b66-a55b-6540dc919704 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:53.590936 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1881928b-a0bc-4b66-a55b-6540dc919704 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:53.591165 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27232a1e-070f-403d-862e-9b836de83457 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:53.591284 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1881928b-a0bc-4b66-a55b-6540dc919704 None admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:53.591930 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1881928b-a0bc-4b66-a55b-6540dc919704 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:53.592245 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1881928b-a0bc-4b66-a55b-6540dc919704 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:53.594279 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1881928b-a0bc-4b66-a55b-6540dc919704 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:53.612329 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 250/512] 10.222.0.25 () {56 vars in 1139 bytes} [Sun Sep 25 04:22:53 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:53.613228 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1881928b-a0bc-4b66-a55b-6540dc919704 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:53.692339 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27232a1e-070f-403d-862e-9b836de83457 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:53.701806 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:53.710850 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1881928b-a0bc-4b66-a55b-6540dc919704 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:53.727764 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b690021-5294-4f28-be25-c9709d169c67 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:53.728399 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b690021-5294-4f28-be25-c9709d169c67 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:53.728705 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b690021-5294-4f28-be25-c9709d169c67 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:53.728980 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b690021-5294-4f28-be25-c9709d169c67 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:53.737133 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b690021-5294-4f28-be25-c9709d169c67 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:53.738582 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b690021-5294-4f28-be25-c9709d169c67 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:53.744906 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1581189-6d9c-4c8d-9641-d0e5c4150c25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:53.745575 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1581189-6d9c-4c8d-9641-d0e5c4150c25 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:53.745857 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1581189-6d9c-4c8d-9641-d0e5c4150c25 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:53.746124 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1581189-6d9c-4c8d-9641-d0e5c4150c25 None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:53.746747 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1581189-6d9c-4c8d-9641-d0e5c4150c25 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:53.747050 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1581189-6d9c-4c8d-9641-d0e5c4150c25 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:53.748421 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1581189-6d9c-4c8d-9641-d0e5c4150c25 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:53.751822 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-8b690021-5294-4f28-be25-c9709d169c67 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 25 04:22:53.751822 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:22:53.751822 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:22:53.751822 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:22:53.751822 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:22:53.751822 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:22:53.751822 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:22:53.751822 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:22:53.751822 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:22:53.751822 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:22:53.751822 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:22:53.751822 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:22:53.751822 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 25 04:22:53.751822 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 25 04:22:53.751822 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 25 04:22:53.751822 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 25 04:22:53.752770 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:22:53.752770 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:22:53.752770 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 25 04:22:53.752770 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 04:22:53.752770 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 25 04:22:53.752770 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 04:22:53.752770 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:22:53.752770 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:22:53.752770 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:22:53.752770 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:22:53.752770 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:22:53.752770 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:22:53.752770 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:22:53.752770 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:22:53.752770 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:22:53.752770 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:22:53.752770 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:22:53.752770 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:22:53.753662 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:22:53.753662 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:22:53.753662 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 25 04:22:53.753662 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 25 04:22:53.753662 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 25 04:22:53.753662 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 25 04:22:53.753662 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 25 04:22:53.753662 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 25 04:22:53.753662 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 25 04:22:53.753662 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:22:53.753662 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 263/513] 10.222.0.25 () {56 vars in 1093 bytes} [Sun Sep 25 04:22:53 2022] GET /identity/v3/users/nova => generated 82 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:22:53.754233 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b690021-5294-4f28-be25-c9709d169c67 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:53.764124 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 251/514] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Sep 25 04:22:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:53.764124 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1581189-6d9c-4c8d-9641-d0e5c4150c25 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:53.806857 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:53.856437 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b690021-5294-4f28-be25-c9709d169c67 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:53.858881 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1581189-6d9c-4c8d-9641-d0e5c4150c25 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:53.890146 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ca06a6a-0a90-45eb-b515-86678f4ca614 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:53.890828 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca06a6a-0a90-45eb-b515-86678f4ca614 None admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:53.891198 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca06a6a-0a90-45eb-b515-86678f4ca614 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:53.891394 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca06a6a-0a90-45eb-b515-86678f4ca614 None admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:53.892034 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ca06a6a-0a90-45eb-b515-86678f4ca614 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:53.892498 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbcf4914-4e91-4aa8-bb45-ac8ca1ee496d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:53.893159 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbcf4914-4e91-4aa8-bb45-ac8ca1ee496d None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:53.893447 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbcf4914-4e91-4aa8-bb45-ac8ca1ee496d None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:53.893540 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ca06a6a-0a90-45eb-b515-86678f4ca614 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:53.893719 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbcf4914-4e91-4aa8-bb45-ac8ca1ee496d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:53.903925 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbcf4914-4e91-4aa8-bb45-ac8ca1ee496d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:53.905776 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbcf4914-4e91-4aa8-bb45-ac8ca1ee496d None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:53.923653 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 264/515] 10.222.0.25 () {60 vars in 1132 bytes} [Sun Sep 25 04:22:53 2022] POST /identity/v3/users => generated 247 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:22:53.923653 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ca06a6a-0a90-45eb-b515-86678f4ca614 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:53.923955 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-dbcf4914-4e91-4aa8-bb45-ac8ca1ee496d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:22:53.923955 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:22:53.923955 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:22:53.923955 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:22:53.923955 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:22:53.923955 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:22:53.923955 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:22:53.923955 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:22:53.923955 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:22:53.923955 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:22:53.923955 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:22:53.923955 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:22:53.923955 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 04:22:53.923955 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 04:22:53.923955 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 04:22:53.923955 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 04:22:53.925026 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:22:53.925026 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:22:53.925026 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:22:53.925026 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:22:53.925026 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:22:53.925026 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:22:53.925026 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:22:53.925026 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:22:53.925026 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:22:53.925026 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:22:53.925026 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:22:53.925026 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:22:53.925026 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:22:53.925026 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:22:53.925026 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:22:53.925026 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:22:53.925026 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 04:22:53.926009 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 04:22:53.926009 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:22:53.926009 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:22:53.926009 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 252/516] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:22:53 2022] GET /identity/v3/domains/Default => generated 87 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:22:53.926009 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbcf4914-4e91-4aa8-bb45-ac8ca1ee496d None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:53.958654 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:54.038432 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:54.044840 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbcf4914-4e91-4aa8-bb45-ac8ca1ee496d None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:54.045964 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ca06a6a-0a90-45eb-b515-86678f4ca614 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:54.083680 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b062d4f9-d7a3-4eed-8d82-c62bbc252e1d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:54.084297 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b062d4f9-d7a3-4eed-8d82-c62bbc252e1d None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:54.084576 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b062d4f9-d7a3-4eed-8d82-c62bbc252e1d None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:54.084875 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b062d4f9-d7a3-4eed-8d82-c62bbc252e1d None admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:54.085604 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b062d4f9-d7a3-4eed-8d82-c62bbc252e1d None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:54.085908 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b062d4f9-d7a3-4eed-8d82-c62bbc252e1d None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:54.086083 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f3ad1c7-5aad-4f68-8a7a-c87fdc04dd57 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:54.086714 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f3ad1c7-5aad-4f68-8a7a-c87fdc04dd57 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:54.086994 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f3ad1c7-5aad-4f68-8a7a-c87fdc04dd57 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:54.087266 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f3ad1c7-5aad-4f68-8a7a-c87fdc04dd57 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:54.087359 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b062d4f9-d7a3-4eed-8d82-c62bbc252e1d None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:54.107126 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 253/517] 10.222.0.25 () {56 vars in 1133 bytes} [Sun Sep 25 04:22:53 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:54.107965 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b062d4f9-d7a3-4eed-8d82-c62bbc252e1d None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:54.173681 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f3ad1c7-5aad-4f68-8a7a-c87fdc04dd57 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:54.175144 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f3ad1c7-5aad-4f68-8a7a-c87fdc04dd57 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:54.204310 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:54.228339 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b062d4f9-d7a3-4eed-8d82-c62bbc252e1d None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:54.264014 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8523a69b-3bb5-4e57-861d-2eb54ad3d10e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:54.264764 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8523a69b-3bb5-4e57-861d-2eb54ad3d10e None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:54.265095 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8523a69b-3bb5-4e57-861d-2eb54ad3d10e None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:54.265367 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8523a69b-3bb5-4e57-861d-2eb54ad3d10e None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:54.273823 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8523a69b-3bb5-4e57-861d-2eb54ad3d10e None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:54.275266 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8523a69b-3bb5-4e57-861d-2eb54ad3d10e None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:54.290868 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 265/518] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Sep 25 04:22:53 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 371 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 25 04:22:54.290868 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f3ad1c7-5aad-4f68-8a7a-c87fdc04dd57 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:54.291105 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-8523a69b-3bb5-4e57-861d-2eb54ad3d10e None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Sep 25 04:22:54.291105 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:22:54.291105 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:22:54.291105 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:22:54.291105 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:22:54.291105 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:22:54.291105 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:22:54.291105 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:22:54.291105 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:22:54.291105 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:22:54.291105 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:22:54.291105 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:22:54.291105 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 25 04:22:54.291105 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 25 04:22:54.291105 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 25 04:22:54.291105 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 25 04:22:54.292003 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:22:54.292003 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:22:54.292003 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 25 04:22:54.292003 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 04:22:54.292003 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 25 04:22:54.292003 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 04:22:54.292003 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:22:54.292003 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:22:54.292003 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:22:54.292003 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:22:54.292003 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:22:54.292003 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:22:54.292003 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:22:54.292003 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:22:54.292003 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:22:54.292003 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:22:54.292003 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:22:54.292003 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:22:54.292949 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:22:54.292949 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:22:54.292949 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 25 04:22:54.292949 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 25 04:22:54.292949 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 25 04:22:54.292949 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 25 04:22:54.292949 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 25 04:22:54.292949 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 25 04:22:54.292949 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Sep 25 04:22:54.292949 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:22:54.293483 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 254/519] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Sep 25 04:22:54 2022] GET /identity/v3/users/cinder => generated 84 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:22:54.294078 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8523a69b-3bb5-4e57-861d-2eb54ad3d10e None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:54.310071 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:54.396200 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f3ad1c7-5aad-4f68-8a7a-c87fdc04dd57 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:54.398948 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8523a69b-3bb5-4e57-861d-2eb54ad3d10e None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:54.423549 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:54.433830 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f51c376-fb87-474b-8a30-ec02bdf62d96 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:54.434443 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f51c376-fb87-474b-8a30-ec02bdf62d96 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:54.434722 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f51c376-fb87-474b-8a30-ec02bdf62d96 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:54.434991 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f51c376-fb87-474b-8a30-ec02bdf62d96 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:54.439985 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99ad0d6a-210c-4cf9-9bd6-a0859dda1f74 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:54.440601 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99ad0d6a-210c-4cf9-9bd6-a0859dda1f74 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:54.440903 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99ad0d6a-210c-4cf9-9bd6-a0859dda1f74 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:54.441173 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99ad0d6a-210c-4cf9-9bd6-a0859dda1f74 None admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:54.441843 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99ad0d6a-210c-4cf9-9bd6-a0859dda1f74 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:54.442143 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99ad0d6a-210c-4cf9-9bd6-a0859dda1f74 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:54.443501 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99ad0d6a-210c-4cf9-9bd6-a0859dda1f74 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:54.457409 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 266/520] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Sep 25 04:22:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:54.459103 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99ad0d6a-210c-4cf9-9bd6-a0859dda1f74 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:54.513761 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f51c376-fb87-474b-8a30-ec02bdf62d96 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:54.515576 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f51c376-fb87-474b-8a30-ec02bdf62d96 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:54.529928 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:54.578592 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99ad0d6a-210c-4cf9-9bd6-a0859dda1f74 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:54.614742 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a795ae11-a96b-42e1-b453-f24028294356 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:54.615377 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a795ae11-a96b-42e1-b453-f24028294356 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:54.620310 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a795ae11-a96b-42e1-b453-f24028294356 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:54.620310 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a795ae11-a96b-42e1-b453-f24028294356 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:54.620505 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 255/521] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Sep 25 04:22:54 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 325 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 25 04:22:54.620505 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f51c376-fb87-474b-8a30-ec02bdf62d96 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:54.631726 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a795ae11-a96b-42e1-b453-f24028294356 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:54.633473 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a795ae11-a96b-42e1-b453-f24028294356 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:54.648262 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-a795ae11-a96b-42e1-b453-f24028294356 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:22:54.648262 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:22:54.648262 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:22:54.648262 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:22:54.648262 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:22:54.648262 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:22:54.648262 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:22:54.648262 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:22:54.648262 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:22:54.648262 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:22:54.648262 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:22:54.648262 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:22:54.648262 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 04:22:54.648262 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 04:22:54.648262 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 04:22:54.648262 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 04:22:54.649456 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:22:54.649456 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:22:54.649456 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:22:54.649456 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:22:54.649456 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:22:54.649456 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:22:54.649456 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:22:54.649456 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:22:54.649456 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:22:54.649456 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:22:54.649456 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:22:54.649456 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:22:54.649456 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:22:54.649456 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:22:54.649456 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:22:54.649456 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:22:54.649456 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 04:22:54.650340 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 04:22:54.650340 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:22:54.650340 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:22:54.650340 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 267/522] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:22:54 2022] GET /identity/v3/domains/Default => generated 87 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:22:54.650555 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a795ae11-a96b-42e1-b453-f24028294356 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:54.694300 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:54.744027 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f51c376-fb87-474b-8a30-ec02bdf62d96 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:54.759022 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:54.767118 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a795ae11-a96b-42e1-b453-f24028294356 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:54.782265 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d79c3c42-e942-42a4-8a3a-e00fad9dbc82 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:54.783081 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d79c3c42-e942-42a4-8a3a-e00fad9dbc82 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:54.783656 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d79c3c42-e942-42a4-8a3a-e00fad9dbc82 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:54.784123 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d79c3c42-e942-42a4-8a3a-e00fad9dbc82 None admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:54.785127 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d79c3c42-e942-42a4-8a3a-e00fad9dbc82 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:54.785593 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d79c3c42-e942-42a4-8a3a-e00fad9dbc82 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:54.787386 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d79c3c42-e942-42a4-8a3a-e00fad9dbc82 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:54.800796 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e18685c6-8053-47e9-820a-f22748f79145 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:54.801424 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e18685c6-8053-47e9-820a-f22748f79145 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:54.801698 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e18685c6-8053-47e9-820a-f22748f79145 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:54.801967 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e18685c6-8053-47e9-820a-f22748f79145 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:54.810374 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 256/523] 10.222.0.25 () {56 vars in 1137 bytes} [Sun Sep 25 04:22:54 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:54.810374 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d79c3c42-e942-42a4-8a3a-e00fad9dbc82 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:54.812723 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e18685c6-8053-47e9-820a-f22748f79145 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:54.814395 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e18685c6-8053-47e9-820a-f22748f79145 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:54.826629 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-e18685c6-8053-47e9-820a-f22748f79145 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 04:22:54.826629 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:22:54.826629 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:22:54.826629 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:22:54.826629 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:22:54.826629 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:22:54.826629 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:22:54.826629 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:22:54.826629 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:22:54.826629 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:22:54.826629 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:22:54.826629 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:22:54.826629 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 25 04:22:54.826629 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 25 04:22:54.826629 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 25 04:22:54.827714 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 25 04:22:54.827714 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:22:54.827714 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:22:54.827714 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:22:54.827714 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:22:54.827714 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:22:54.827714 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:22:54.827714 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:22:54.827714 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:22:54.827714 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:22:54.827714 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:22:54.827714 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:22:54.827714 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:22:54.827714 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:22:54.827714 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:22:54.827714 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:22:54.827714 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:22:54.834059 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 25 04:22:54.834059 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 25 04:22:54.834059 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 25 04:22:54.834059 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 25 04:22:54.834059 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 25 04:22:54.834059 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 25 04:22:54.834059 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 04:22:54.834059 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:22:54.834059 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 268/524] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Sep 25 04:22:54 2022] GET /identity/v3/projects/service => generated 88 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:22:54.834059 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e18685c6-8053-47e9-820a-f22748f79145 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:54.921599 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d79c3c42-e942-42a4-8a3a-e00fad9dbc82 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:54.934909 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:54.942618 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e18685c6-8053-47e9-820a-f22748f79145 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:54.957330 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd9bb46e-e372-4de0-8e01-79c3d798b812 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:54.957964 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd9bb46e-e372-4de0-8e01-79c3d798b812 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:54.958245 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd9bb46e-e372-4de0-8e01-79c3d798b812 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:54.958519 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd9bb46e-e372-4de0-8e01-79c3d798b812 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:54.970288 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd9bb46e-e372-4de0-8e01-79c3d798b812 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:54.972415 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd9bb46e-e372-4de0-8e01-79c3d798b812 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:54.978280 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b664621f-a2d1-4832-ad67-496ce469107d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:54.978948 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b664621f-a2d1-4832-ad67-496ce469107d None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:54.979295 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b664621f-a2d1-4832-ad67-496ce469107d None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:54.979629 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b664621f-a2d1-4832-ad67-496ce469107d None admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:54.980328 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b664621f-a2d1-4832-ad67-496ce469107d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:54.980730 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b664621f-a2d1-4832-ad67-496ce469107d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:54.982149 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b664621f-a2d1-4832-ad67-496ce469107d None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:54.989472 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-cd9bb46e-e372-4de0-8e01-79c3d798b812 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:22:54.989472 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:22:54.989472 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:22:54.989472 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:22:54.989472 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:22:54.989472 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:22:54.989472 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:22:54.989472 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:22:54.989472 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:22:54.989472 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:22:54.989472 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:22:54.989472 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:22:54.989472 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 04:22:54.989472 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 04:22:54.989472 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 04:22:54.989472 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 04:22:54.990610 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:22:54.990610 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:22:54.990610 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:22:54.990610 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:22:54.990610 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:22:54.990610 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:22:54.990610 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:22:54.990610 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:22:54.990610 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:22:54.990610 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:22:54.990610 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:22:54.990610 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:22:54.990610 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:22:54.990610 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:22:54.990610 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:22:54.990610 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:22:54.990610 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 04:22:54.991735 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 04:22:54.991735 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:22:54.991735 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:22:54.992297 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 257/525] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:22:54 2022] GET /identity/v3/domains/Default => generated 87 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:22:54.993229 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd9bb46e-e372-4de0-8e01-79c3d798b812 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:54.998257 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 269/526] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Sep 25 04:22:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:54.999237 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b664621f-a2d1-4832-ad67-496ce469107d None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:55.021737 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:55.102541 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd9bb46e-e372-4de0-8e01-79c3d798b812 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:55.112461 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b664621f-a2d1-4832-ad67-496ce469107d None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:55.138904 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcf92c13-650b-4a1d-98cc-a8ba12527748 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:55.139831 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf92c13-650b-4a1d-98cc-a8ba12527748 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:55.140120 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf92c13-650b-4a1d-98cc-a8ba12527748 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:55.141334 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf92c13-650b-4a1d-98cc-a8ba12527748 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:55.151109 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5e27ce7-d2b8-4686-99e3-fa8bb00e9ea5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:55.151825 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5e27ce7-d2b8-4686-99e3-fa8bb00e9ea5 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:55.152195 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5e27ce7-d2b8-4686-99e3-fa8bb00e9ea5 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:55.152477 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5e27ce7-d2b8-4686-99e3-fa8bb00e9ea5 None admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:55.153189 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5e27ce7-d2b8-4686-99e3-fa8bb00e9ea5 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:55.153492 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5e27ce7-d2b8-4686-99e3-fa8bb00e9ea5 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:55.154911 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5e27ce7-d2b8-4686-99e3-fa8bb00e9ea5 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:55.178380 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 270/527] 10.222.0.25 () {56 vars in 1148 bytes} [Sun Sep 25 04:22:54 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:55.179620 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5e27ce7-d2b8-4686-99e3-fa8bb00e9ea5 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:55.199587 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:55.222520 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcf92c13-650b-4a1d-98cc-a8ba12527748 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:55.224024 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcf92c13-650b-4a1d-98cc-a8ba12527748 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:55.253126 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:55.286668 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5e27ce7-d2b8-4686-99e3-fa8bb00e9ea5 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:55.325246 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb15e711-fd76-411c-89ab-921fdff702a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:55.326117 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb15e711-fd76-411c-89ab-921fdff702a2 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:55.326397 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb15e711-fd76-411c-89ab-921fdff702a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:55.326670 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb15e711-fd76-411c-89ab-921fdff702a2 None admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:55.327311 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb15e711-fd76-411c-89ab-921fdff702a2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:55.327614 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb15e711-fd76-411c-89ab-921fdff702a2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:55.329623 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb15e711-fd76-411c-89ab-921fdff702a2 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:55.338392 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 258/528] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Sep 25 04:22:54 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 346 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 25 04:22:55.339478 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcf92c13-650b-4a1d-98cc-a8ba12527748 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:55.346278 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 271/529] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Sep 25 04:22:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:55.347100 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb15e711-fd76-411c-89ab-921fdff702a2 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:55.431454 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:55.445747 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcf92c13-650b-4a1d-98cc-a8ba12527748 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:55.462290 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb15e711-fd76-411c-89ab-921fdff702a2 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:55.473153 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:55.489357 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc7ee479-eb0c-4b22-97d9-63d3db3c9a44 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:55.490237 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc7ee479-eb0c-4b22-97d9-63d3db3c9a44 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:55.490691 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc7ee479-eb0c-4b22-97d9-63d3db3c9a44 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:55.491034 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc7ee479-eb0c-4b22-97d9-63d3db3c9a44 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:55.498059 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc7ee479-eb0c-4b22-97d9-63d3db3c9a44 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:55.499519 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc7ee479-eb0c-4b22-97d9-63d3db3c9a44 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:55.501799 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2a58e28-77b8-48eb-bc95-73b36f9cb5c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:55.502411 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a58e28-77b8-48eb-bc95-73b36f9cb5c8 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:55.502693 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a58e28-77b8-48eb-bc95-73b36f9cb5c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:55.502967 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a58e28-77b8-48eb-bc95-73b36f9cb5c8 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:55.503734 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2a58e28-77b8-48eb-bc95-73b36f9cb5c8 None admin] RBAC: Adding query filter params (role.id=9b44520f482c4423bd10568d0f86718c, scope.project.id=056ca1b1a9974a7581550de0d9217823, user.id=316eaa67d5464091af47efd489b96b87) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:55.503997 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2a58e28-77b8-48eb-bc95-73b36f9cb5c8 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:55.505562 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2a58e28-77b8-48eb-bc95-73b36f9cb5c8 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:55.520813 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 272/530] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Sep 25 04:22:55 2022] GET /identity/v3/role_assignments?user.id=316eaa67d5464091af47efd489b96b87&scope.project.id=056ca1b1a9974a7581550de0d9217823&role.id=9b44520f482c4423bd10568d0f86718c => generated 259 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:55.520813 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2a58e28-77b8-48eb-bc95-73b36f9cb5c8 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:55.521088 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-fc7ee479-eb0c-4b22-97d9-63d3db3c9a44 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 04:22:55.521088 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:22:55.521088 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:22:55.521088 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:22:55.521088 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:22:55.521088 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:22:55.521088 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:22:55.521088 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:22:55.521088 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:22:55.521088 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:22:55.521088 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:22:55.521088 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:22:55.521088 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 25 04:22:55.521088 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 25 04:22:55.521088 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 25 04:22:55.522097 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 25 04:22:55.522097 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:22:55.522097 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:22:55.522097 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:22:55.522097 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:22:55.522097 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:22:55.522097 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:22:55.522097 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:22:55.522097 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:22:55.522097 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:22:55.522097 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:22:55.522097 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:22:55.522097 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:22:55.522097 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:22:55.522097 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:22:55.522097 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:22:55.522097 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:22:55.523190 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 25 04:22:55.523190 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 25 04:22:55.523190 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 25 04:22:55.523190 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 25 04:22:55.523190 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 25 04:22:55.523190 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 25 04:22:55.523190 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 04:22:55.523190 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:22:55.523190 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 259/531] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Sep 25 04:22:55 2022] GET /identity/v3/projects/service => generated 88 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:22:55.523190 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc7ee479-eb0c-4b22-97d9-63d3db3c9a44 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:55.641457 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2a58e28-77b8-48eb-bc95-73b36f9cb5c8 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:55.666332 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc7ee479-eb0c-4b22-97d9-63d3db3c9a44 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:55.684801 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26ed221e-6597-4819-948f-cbfaf2a98299 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:55.685547 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26ed221e-6597-4819-948f-cbfaf2a98299 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:55.685896 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26ed221e-6597-4819-948f-cbfaf2a98299 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:55.686231 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26ed221e-6597-4819-948f-cbfaf2a98299 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:55.697486 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:55.700903 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26ed221e-6597-4819-948f-cbfaf2a98299 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:55.702883 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26ed221e-6597-4819-948f-cbfaf2a98299 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:55.717738 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abcab9d0-b1dc-404b-97a6-21e1fc8790cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:55.718381 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abcab9d0-b1dc-404b-97a6-21e1fc8790cb None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:55.719226 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abcab9d0-b1dc-404b-97a6-21e1fc8790cb None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:55.719670 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abcab9d0-b1dc-404b-97a6-21e1fc8790cb None admin] PATH_INFO: `/v3/projects/service` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:55.720462 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-26ed221e-6597-4819-948f-cbfaf2a98299 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:22:55.720462 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:22:55.720462 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:22:55.720462 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:22:55.720462 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:22:55.720462 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:22:55.720462 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:22:55.720462 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:22:55.720462 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:22:55.720462 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:22:55.720462 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:22:55.720462 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:22:55.720462 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 04:22:55.720462 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 04:22:55.720462 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 04:22:55.720462 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 04:22:55.721415 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:22:55.721415 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:22:55.721415 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:22:55.721415 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:22:55.721415 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:22:55.721415 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:22:55.721415 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:22:55.721415 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:22:55.721415 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:22:55.721415 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:22:55.721415 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:22:55.721415 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:22:55.721415 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:22:55.721415 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:22:55.721415 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:22:55.721415 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:22:55.721415 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 04:22:55.722283 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 04:22:55.722283 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:22:55.722283 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:22:55.722283 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 273/532] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:22:55 2022] GET /identity/v3/domains/Default => generated 87 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:22:55.722934 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26ed221e-6597-4819-948f-cbfaf2a98299 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:55.731559 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abcab9d0-b1dc-404b-97a6-21e1fc8790cb None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:55.733064 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abcab9d0-b1dc-404b-97a6-21e1fc8790cb None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:55.746298 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-abcab9d0-b1dc-404b-97a6-21e1fc8790cb None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 04:22:55.746298 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:22:55.746298 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:22:55.746298 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:22:55.746298 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:22:55.746298 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:22:55.746298 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:22:55.746298 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:22:55.746298 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:22:55.746298 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:22:55.746298 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:22:55.746298 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:22:55.746298 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 25 04:22:55.746298 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 25 04:22:55.746298 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 25 04:22:55.747208 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 25 04:22:55.747208 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:22:55.747208 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:22:55.747208 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:22:55.747208 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:22:55.747208 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:22:55.747208 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:22:55.747208 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:22:55.747208 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:22:55.747208 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:22:55.747208 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:22:55.747208 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:22:55.747208 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:22:55.747208 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:22:55.747208 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:22:55.747208 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:22:55.747208 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:22:55.748078 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 25 04:22:55.748078 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 25 04:22:55.748078 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 25 04:22:55.748078 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 25 04:22:55.748078 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 25 04:22:55.748078 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 25 04:22:55.748078 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 04:22:55.748078 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:22:55.752324 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 260/533] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Sep 25 04:22:55 2022] GET /identity/v3/projects/service => generated 88 bytes in 234 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:22:55.753367 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abcab9d0-b1dc-404b-97a6-21e1fc8790cb None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:55.818112 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:55.838083 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26ed221e-6597-4819-948f-cbfaf2a98299 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:55.882941 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b97804a9-1c07-4f77-9c70-b3a66fc920c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:55.884573 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b97804a9-1c07-4f77-9c70-b3a66fc920c0 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:55.885256 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b97804a9-1c07-4f77-9c70-b3a66fc920c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:55.885845 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b97804a9-1c07-4f77-9c70-b3a66fc920c0 None admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:55.886987 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b97804a9-1c07-4f77-9c70-b3a66fc920c0 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:55.887855 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b97804a9-1c07-4f77-9c70-b3a66fc920c0 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:55.889782 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b97804a9-1c07-4f77-9c70-b3a66fc920c0 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:55.890233 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abcab9d0-b1dc-404b-97a6-21e1fc8790cb None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:55.907743 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 274/534] 10.222.0.25 () {56 vars in 1148 bytes} [Sun Sep 25 04:22:55 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:55.908921 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b97804a9-1c07-4f77-9c70-b3a66fc920c0 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:55.932444 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dde0974-b438-46a5-a28a-b889a9fcd71a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:55.932865 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dde0974-b438-46a5-a28a-b889a9fcd71a None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:55.933190 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dde0974-b438-46a5-a28a-b889a9fcd71a None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:55.933466 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dde0974-b438-46a5-a28a-b889a9fcd71a None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:55.934100 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dde0974-b438-46a5-a28a-b889a9fcd71a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:55.934420 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dde0974-b438-46a5-a28a-b889a9fcd71a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:55.935766 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dde0974-b438-46a5-a28a-b889a9fcd71a None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:55.954159 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 261/535] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Sep 25 04:22:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:55.954159 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dde0974-b438-46a5-a28a-b889a9fcd71a None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:55.989196 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:56.052106 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b97804a9-1c07-4f77-9c70-b3a66fc920c0 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:56.065794 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dde0974-b438-46a5-a28a-b889a9fcd71a None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:56.068367 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:56.096533 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70260b3e-7096-4efc-977a-08862c8a126f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:56.097327 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70260b3e-7096-4efc-977a-08862c8a126f None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:56.097736 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70260b3e-7096-4efc-977a-08862c8a126f None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:56.098106 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70260b3e-7096-4efc-977a-08862c8a126f None admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:56.099068 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70260b3e-7096-4efc-977a-08862c8a126f None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:56.099776 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70260b3e-7096-4efc-977a-08862c8a126f None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:56.101616 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70260b3e-7096-4efc-977a-08862c8a126f None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:56.107298 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eae52b78-2f9c-46e7-9858-ab1707ad32ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:56.107913 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eae52b78-2f9c-46e7-9858-ab1707ad32ec None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:56.108191 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eae52b78-2f9c-46e7-9858-ab1707ad32ec None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:56.108465 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eae52b78-2f9c-46e7-9858-ab1707ad32ec None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:56.109252 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eae52b78-2f9c-46e7-9858-ab1707ad32ec None admin] RBAC: Adding query filter params (role.id=9b44520f482c4423bd10568d0f86718c, scope.project.id=056ca1b1a9974a7581550de0d9217823, user.id=0268e88e0c5645a98bab5664be6e49bc) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:56.110263 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eae52b78-2f9c-46e7-9858-ab1707ad32ec None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:56.111941 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eae52b78-2f9c-46e7-9858-ab1707ad32ec None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:56.120868 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 275/536] 10.222.0.25 () {56 vars in 1148 bytes} [Sun Sep 25 04:22:55 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:56.121887 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70260b3e-7096-4efc-977a-08862c8a126f None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:56.128906 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 262/537] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Sep 25 04:22:55 2022] GET /identity/v3/role_assignments?user.id=0268e88e0c5645a98bab5664be6e49bc&scope.project.id=056ca1b1a9974a7581550de0d9217823&role.id=9b44520f482c4423bd10568d0f86718c => generated 259 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:56.130003 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eae52b78-2f9c-46e7-9858-ab1707ad32ec None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:56.243671 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70260b3e-7096-4efc-977a-08862c8a126f None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:56.254547 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eae52b78-2f9c-46e7-9858-ab1707ad32ec None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:56.260148 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:56.289268 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a7c1844-dcf6-4064-8e13-ddef187dd757 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:56.290088 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a7c1844-dcf6-4064-8e13-ddef187dd757 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:56.290472 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a7c1844-dcf6-4064-8e13-ddef187dd757 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:56.290845 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a7c1844-dcf6-4064-8e13-ddef187dd757 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:56.297402 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f57670b4-b1a2-4818-adcf-dbe2f4b144bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:56.298325 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f57670b4-b1a2-4818-adcf-dbe2f4b144bf None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:56.298746 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f57670b4-b1a2-4818-adcf-dbe2f4b144bf None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:56.299157 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f57670b4-b1a2-4818-adcf-dbe2f4b144bf None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:56.299317 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a7c1844-dcf6-4064-8e13-ddef187dd757 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:56.300185 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f57670b4-b1a2-4818-adcf-dbe2f4b144bf None admin] RBAC: Adding query filter params (role.id=9b44520f482c4423bd10568d0f86718c, scope.project.id=056ca1b1a9974a7581550de0d9217823, user.id=e703b42c53974daaa608b0ccc3f018b1) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:56.300728 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f57670b4-b1a2-4818-adcf-dbe2f4b144bf None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:56.301122 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a7c1844-dcf6-4064-8e13-ddef187dd757 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:56.302698 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f57670b4-b1a2-4818-adcf-dbe2f4b144bf None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:56.309345 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 263/538] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Sep 25 04:22:56 2022] GET /identity/v3/role_assignments?user.id=e703b42c53974daaa608b0ccc3f018b1&scope.project.id=056ca1b1a9974a7581550de0d9217823&role.id=9b44520f482c4423bd10568d0f86718c => generated 259 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:56.310949 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85f020c5-585a-4a52-8f0c-8178ebb77fe7 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:56.311360 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85f020c5-585a-4a52-8f0c-8178ebb77fe7 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:56.311759 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85f020c5-585a-4a52-8f0c-8178ebb77fe7 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:56.313487 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 264/539] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:22:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:22:56.319944 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9848ab9-1376-47a2-885a-4a94991873f8 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:56.320416 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9848ab9-1376-47a2-885a-4a94991873f8 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:56.320908 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9848ab9-1376-47a2-885a-4a94991873f8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:56.327643 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-0a7c1844-dcf6-4064-8e13-ddef187dd757 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 04:22:56.327643 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:22:56.327643 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:22:56.327643 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:22:56.327643 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:22:56.327643 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:22:56.327643 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:22:56.327643 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:22:56.327643 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:22:56.327643 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:22:56.327643 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:22:56.327643 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:22:56.327643 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 25 04:22:56.327643 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 25 04:22:56.327643 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 25 04:22:56.329817 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 25 04:22:56.329817 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:22:56.329817 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:22:56.329817 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:22:56.329817 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:22:56.329817 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:22:56.329817 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:22:56.329817 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:22:56.329817 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:22:56.329817 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:22:56.329817 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:22:56.329817 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:22:56.329817 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:22:56.329817 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:22:56.329817 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:22:56.329817 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:22:56.329817 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:22:56.330872 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 25 04:22:56.330872 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 25 04:22:56.330872 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 25 04:22:56.330872 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 25 04:22:56.330872 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 25 04:22:56.330872 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 25 04:22:56.330872 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 04:22:56.330872 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:22:56.332312 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 276/540] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Sep 25 04:22:56 2022] GET /identity/v3/projects/service => generated 88 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:22:56.337857 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a7c1844-dcf6-4064-8e13-ddef187dd757 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:56.342323 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-a9848ab9-1376-47a2-885a-4a94991873f8 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:22:56.343568 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-a9848ab9-1376-47a2-885a-4a94991873f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:22:56.393113 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:56.461275 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 265/541] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:22:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 142 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 04:22:56.544909 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a7c1844-dcf6-4064-8e13-ddef187dd757 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:56.580247 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a185454-3c6b-4bc2-a257-c87e11d558e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:56.580823 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a185454-3c6b-4bc2-a257-c87e11d558e4 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:56.581104 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a185454-3c6b-4bc2-a257-c87e11d558e4 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:56.581377 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a185454-3c6b-4bc2-a257-c87e11d558e4 None admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:56.582066 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a185454-3c6b-4bc2-a257-c87e11d558e4 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:56.582374 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a185454-3c6b-4bc2-a257-c87e11d558e4 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:56.583806 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a185454-3c6b-4bc2-a257-c87e11d558e4 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:56.596760 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 277/542] 10.222.0.25 () {56 vars in 1148 bytes} [Sun Sep 25 04:22:56 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 259 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:56.602629 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9848ab9-1376-47a2-885a-4a94991873f8 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:56.752325 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9848ab9-1376-47a2-885a-4a94991873f8 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:56.776375 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a185454-3c6b-4bc2-a257-c87e11d558e4 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:56.809395 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef07a4be-1cf1-4c92-9557-4f9748f2f131 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:56.810364 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef07a4be-1cf1-4c92-9557-4f9748f2f131 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:56.810866 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef07a4be-1cf1-4c92-9557-4f9748f2f131 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:56.811471 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef07a4be-1cf1-4c92-9557-4f9748f2f131 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:56.813498 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef07a4be-1cf1-4c92-9557-4f9748f2f131 None admin] RBAC: Adding query filter params (role.id=9b44520f482c4423bd10568d0f86718c, scope.project.id=056ca1b1a9974a7581550de0d9217823, user.id=ffbc136c77b84e8c87128f7247f4ec03) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:56.814235 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef07a4be-1cf1-4c92-9557-4f9748f2f131 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:56.817026 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef07a4be-1cf1-4c92-9557-4f9748f2f131 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:56.831208 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:56.832236 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 266/543] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Sep 25 04:22:56 2022] GET /identity/v3/role_assignments?user.id=ffbc136c77b84e8c87128f7247f4ec03&scope.project.id=056ca1b1a9974a7581550de0d9217823&role.id=9b44520f482c4423bd10568d0f86718c => generated 259 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:56.933452 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a185454-3c6b-4bc2-a257-c87e11d558e4 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:56.989551 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3342615a-03e1-4300-a990-b1b596bd65bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:56.990410 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3342615a-03e1-4300-a990-b1b596bd65bf None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:56.990794 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3342615a-03e1-4300-a990-b1b596bd65bf None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:56.991189 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3342615a-03e1-4300-a990-b1b596bd65bf None admin] PATH_INFO: `/v3/roles/service` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:57.003802 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3342615a-03e1-4300-a990-b1b596bd65bf None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 04:22:57.009098 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-3342615a-03e1-4300-a990-b1b596bd65bf None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 25 04:22:57.009098 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:22:57.009098 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:22:57.009098 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:22:57.009098 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:22:57.009098 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:22:57.009098 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:22:57.009098 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:22:57.009098 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:22:57.009098 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:22:57.009098 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:22:57.009098 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:22:57.009098 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 04:22:57.009098 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 04:22:57.009098 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 04:22:57.009098 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 04:22:57.010201 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 04:22:57.010201 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 04:22:57.010201 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 04:22:57.010201 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 04:22:57.010201 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:22:57.010201 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:22:57.010201 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:22:57.010201 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:22:57.010201 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:22:57.010201 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:22:57.010201 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:22:57.010201 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:22:57.010201 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:22:57.010201 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:22:57.010201 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:22:57.010201 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:22:57.011189 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:22:57.011189 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:22:57.011189 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:22:57.011189 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:22:57.011189 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 04:22:57.011189 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 04:22:57.011189 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 04:22:57.011189 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 04:22:57.011189 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 04:22:57.011189 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 04:22:57.011189 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 25 04:22:57.011189 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:22:57.016657 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 278/544] 10.222.0.25 () {56 vars in 1102 bytes} [Sun Sep 25 04:22:56 2022] GET /identity/v3/roles/service => generated 85 bytes in 237 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 04:22:57.020833 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef07a4be-1cf1-4c92-9557-4f9748f2f131 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:57.050620 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:57.135375 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef07a4be-1cf1-4c92-9557-4f9748f2f131 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:57.171803 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24a40cc0-d6b6-44ae-8cae-b1d735a92221 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:57.172589 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24a40cc0-d6b6-44ae-8cae-b1d735a92221 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:57.172997 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24a40cc0-d6b6-44ae-8cae-b1d735a92221 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:57.173441 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24a40cc0-d6b6-44ae-8cae-b1d735a92221 None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:57.174201 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24a40cc0-d6b6-44ae-8cae-b1d735a92221 None admin] RBAC: Adding query filter params (name=service) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:57.174632 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24a40cc0-d6b6-44ae-8cae-b1d735a92221 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:57.176016 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24a40cc0-d6b6-44ae-8cae-b1d735a92221 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:57.184902 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 267/545] 10.222.0.25 () {56 vars in 1103 bytes} [Sun Sep 25 04:22:57 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:57.191219 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3342615a-03e1-4300-a990-b1b596bd65bf None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:57.304151 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:57.314272 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3342615a-03e1-4300-a990-b1b596bd65bf None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:57.369031 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a360ca97-15f1-40a7-ae84-233132e09c6e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:57.369670 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a360ca97-15f1-40a7-ae84-233132e09c6e None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:57.369952 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a360ca97-15f1-40a7-ae84-233132e09c6e None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:57.370279 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a360ca97-15f1-40a7-ae84-233132e09c6e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:57.386005 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a360ca97-15f1-40a7-ae84-233132e09c6e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:57.387396 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a360ca97-15f1-40a7-ae84-233132e09c6e None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:57.410113 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-a360ca97-15f1-40a7-ae84-233132e09c6e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:22:57.410113 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:22:57.410113 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:22:57.410113 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:22:57.410113 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:22:57.410113 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:22:57.410113 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:22:57.410113 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:22:57.410113 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:22:57.410113 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:22:57.410113 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:22:57.410113 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:22:57.410113 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 04:22:57.410113 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 04:22:57.410113 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 04:22:57.410113 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 04:22:57.411108 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:22:57.411108 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:22:57.411108 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:22:57.411108 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:22:57.411108 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:22:57.411108 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:22:57.411108 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:22:57.411108 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:22:57.411108 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:22:57.411108 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:22:57.411108 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:22:57.411108 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:22:57.411108 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:22:57.411108 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:22:57.411108 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:22:57.411108 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:22:57.411108 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 04:22:57.411984 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 04:22:57.411984 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:22:57.411984 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:22:57.411984 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 279/546] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:22:57 2022] GET /identity/v3/domains/Default => generated 87 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:22:57.416871 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24a40cc0-d6b6-44ae-8cae-b1d735a92221 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:57.545627 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:57.560670 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24a40cc0-d6b6-44ae-8cae-b1d735a92221 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:57.605140 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8ab4363-a0fd-408b-82e2-38ea3c6c6646 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:57.605756 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8ab4363-a0fd-408b-82e2-38ea3c6c6646 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:57.606038 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8ab4363-a0fd-408b-82e2-38ea3c6c6646 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:57.606310 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8ab4363-a0fd-408b-82e2-38ea3c6c6646 None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:57.606952 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8ab4363-a0fd-408b-82e2-38ea3c6c6646 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:57.607256 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8ab4363-a0fd-408b-82e2-38ea3c6c6646 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:57.608703 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8ab4363-a0fd-408b-82e2-38ea3c6c6646 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:57.623866 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 268/547] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Sep 25 04:22:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:57.629516 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a360ca97-15f1-40a7-ae84-233132e09c6e None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:57.760715 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a360ca97-15f1-40a7-ae84-233132e09c6e None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:57.800616 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b98e8232-f7f4-4b5b-a9b7-35a1412e46e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:57.801271 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b98e8232-f7f4-4b5b-a9b7-35a1412e46e6 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:57.801611 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b98e8232-f7f4-4b5b-a9b7-35a1412e46e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:57.801891 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b98e8232-f7f4-4b5b-a9b7-35a1412e46e6 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:57.813417 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b98e8232-f7f4-4b5b-a9b7-35a1412e46e6 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:57.814899 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b98e8232-f7f4-4b5b-a9b7-35a1412e46e6 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:57.834304 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-b98e8232-f7f4-4b5b-a9b7-35a1412e46e6 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 25 04:22:57.834304 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:22:57.834304 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:22:57.834304 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:22:57.834304 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:22:57.834304 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:22:57.834304 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:22:57.834304 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:22:57.834304 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:22:57.834304 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:22:57.834304 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:22:57.834304 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:22:57.834304 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 25 04:22:57.834304 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 25 04:22:57.834304 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 25 04:22:57.834304 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 25 04:22:57.835793 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:22:57.835793 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:22:57.835793 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 25 04:22:57.835793 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 04:22:57.835793 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 25 04:22:57.835793 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 04:22:57.835793 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:22:57.835793 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:22:57.835793 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:22:57.835793 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:22:57.835793 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:22:57.835793 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:22:57.835793 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:22:57.835793 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:22:57.835793 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:22:57.835793 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:22:57.835793 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:22:57.835793 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:22:57.836922 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:22:57.836922 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:22:57.836922 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 25 04:22:57.836922 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 25 04:22:57.836922 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 25 04:22:57.836922 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 25 04:22:57.836922 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 25 04:22:57.836922 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 25 04:22:57.836922 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 25 04:22:57.836922 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:22:57.836922 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 280/548] 10.222.0.25 () {56 vars in 1096 bytes} [Sun Sep 25 04:22:57 2022] GET /identity/v3/users/swift => generated 83 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:22:57.841426 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8ab4363-a0fd-408b-82e2-38ea3c6c6646 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:57.984586 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8ab4363-a0fd-408b-82e2-38ea3c6c6646 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:58.029976 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3486c9de-f103-4aae-8a51-0558e673df25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:58.030582 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3486c9de-f103-4aae-8a51-0558e673df25 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:58.030862 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3486c9de-f103-4aae-8a51-0558e673df25 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:58.031710 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3486c9de-f103-4aae-8a51-0558e673df25 None admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:58.032914 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3486c9de-f103-4aae-8a51-0558e673df25 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:58.033224 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3486c9de-f103-4aae-8a51-0558e673df25 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:58.034673 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3486c9de-f103-4aae-8a51-0558e673df25 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:58.064693 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 269/549] 10.222.0.25 () {56 vars in 1135 bytes} [Sun Sep 25 04:22:57 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:58.070573 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b98e8232-f7f4-4b5b-a9b7-35a1412e46e6 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:58.078281 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:58.231334 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b98e8232-f7f4-4b5b-a9b7-35a1412e46e6 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:58.274221 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6ce4c62-40a3-43b3-a318-26720d1b2f3f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:58.274835 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6ce4c62-40a3-43b3-a318-26720d1b2f3f None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:58.275113 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6ce4c62-40a3-43b3-a318-26720d1b2f3f None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:58.275387 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6ce4c62-40a3-43b3-a318-26720d1b2f3f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:58.371168 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92140013-ce54-44c5-9dad-3bda0f379581 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:58.371470 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92140013-ce54-44c5-9dad-3bda0f379581 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:58.371573 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6ce4c62-40a3-43b3-a318-26720d1b2f3f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:58.371671 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92140013-ce54-44c5-9dad-3bda0f379581 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:58.372871 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6ce4c62-40a3-43b3-a318-26720d1b2f3f None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:58.375684 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 270/550] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:22:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:22:58.380202 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:58.380332 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-528f7518-7b9b-42a9-82e3-1e4bcc061570 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:58.380332 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-528f7518-7b9b-42a9-82e3-1e4bcc061570 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:58.380508 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-528f7518-7b9b-42a9-82e3-1e4bcc061570 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:58.401092 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:58.417667 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-528f7518-7b9b-42a9-82e3-1e4bcc061570 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:22:58.418902 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-528f7518-7b9b-42a9-82e3-1e4bcc061570 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:22:58.497810 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 281/551] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Sep 25 04:22:58 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 428 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 25 04:22:58.504593 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6ce4c62-40a3-43b3-a318-26720d1b2f3f None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:58.566698 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 271/552] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:22:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 04:22:58.568338 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c65fb42-2a38-4200-ad7b-64f190f767a7 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:58.569005 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c65fb42-2a38-4200-ad7b-64f190f767a7 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:58.569509 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c65fb42-2a38-4200-ad7b-64f190f767a7 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:58.574800 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 272/553] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:22:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:22:58.578694 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a5cdbfe-35fd-4f7a-b3b5-cf256d66de9a None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:58.581647 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a5cdbfe-35fd-4f7a-b3b5-cf256d66de9a None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:58.582140 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a5cdbfe-35fd-4f7a-b3b5-cf256d66de9a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:58.621527 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-2a5cdbfe-35fd-4f7a-b3b5-cf256d66de9a None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:22:58.623057 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-2a5cdbfe-35fd-4f7a-b3b5-cf256d66de9a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:22:58.690538 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6ce4c62-40a3-43b3-a318-26720d1b2f3f None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:58.692922 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:58.751715 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05614633-a09c-4046-b96c-f2875b19f378 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:58.752360 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05614633-a09c-4046-b96c-f2875b19f378 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:58.752654 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05614633-a09c-4046-b96c-f2875b19f378 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:58.752931 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05614633-a09c-4046-b96c-f2875b19f378 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:58.773820 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 273/554] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:22:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 196 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 04:22:58.774602 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05614633-a09c-4046-b96c-f2875b19f378 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:58.776092 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05614633-a09c-4046-b96c-f2875b19f378 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:58.792049 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-05614633-a09c-4046-b96c-f2875b19f378 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:22:58.792049 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:22:58.792049 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:22:58.792049 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:22:58.792049 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:22:58.792049 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:22:58.792049 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:22:58.792049 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:22:58.792049 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:22:58.792049 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:22:58.792049 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:22:58.792049 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:22:58.792049 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 04:22:58.792049 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 04:22:58.792049 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 04:22:58.792049 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 04:22:58.793087 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:22:58.793087 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:22:58.793087 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:22:58.793087 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:22:58.793087 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:22:58.793087 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:22:58.793087 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:22:58.793087 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:22:58.793087 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:22:58.793087 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:22:58.793087 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:22:58.793087 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:22:58.793087 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:22:58.793087 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:22:58.793087 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:22:58.793087 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:22:58.793087 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 04:22:58.793985 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 04:22:58.793985 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:22:58.793985 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:22:58.793985 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 282/555] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:22:58 2022] GET /identity/v3/domains/Default => generated 87 bytes in 289 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:22:58.801799 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a5cdbfe-35fd-4f7a-b3b5-cf256d66de9a None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:58.868876 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05614633-a09c-4046-b96c-f2875b19f378 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:58.916993 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:58.976452 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a5cdbfe-35fd-4f7a-b3b5-cf256d66de9a None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:59.038816 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05614633-a09c-4046-b96c-f2875b19f378 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:59.054721 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a8c5190-b0af-4c4a-82c0-cd4b410ff1a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:59.055347 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a8c5190-b0af-4c4a-82c0-cd4b410ff1a7 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:59.055626 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a8c5190-b0af-4c4a-82c0-cd4b410ff1a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:59.055897 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a8c5190-b0af-4c4a-82c0-cd4b410ff1a7 None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:59.056542 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a8c5190-b0af-4c4a-82c0-cd4b410ff1a7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:59.056878 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a8c5190-b0af-4c4a-82c0-cd4b410ff1a7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:59.058272 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a8c5190-b0af-4c4a-82c0-cd4b410ff1a7 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:59.070629 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 274/556] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Sep 25 04:22:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 269 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:59.076350 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a8c5190-b0af-4c4a-82c0-cd4b410ff1a7 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:59.088010 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8fd2bb1-453a-45ba-8e08-cab54b6b5a32 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:59.088704 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8fd2bb1-453a-45ba-8e08-cab54b6b5a32 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:59.088997 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8fd2bb1-453a-45ba-8e08-cab54b6b5a32 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:59.089269 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8fd2bb1-453a-45ba-8e08-cab54b6b5a32 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:59.110784 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8fd2bb1-453a-45ba-8e08-cab54b6b5a32 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 04:22:59.117270 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-e8fd2bb1-453a-45ba-8e08-cab54b6b5a32 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 25 04:22:59.117270 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:22:59.117270 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:22:59.117270 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:22:59.117270 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:22:59.117270 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:22:59.117270 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:22:59.117270 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:22:59.117270 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:22:59.117270 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:22:59.117270 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:22:59.117270 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:22:59.117270 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 04:22:59.117270 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 04:22:59.117270 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 04:22:59.117270 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 04:22:59.118210 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 04:22:59.118210 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 04:22:59.118210 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 04:22:59.118210 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 04:22:59.118210 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:22:59.118210 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:22:59.118210 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:22:59.118210 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:22:59.118210 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:22:59.118210 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:22:59.118210 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:22:59.118210 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:22:59.118210 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:22:59.118210 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:22:59.118210 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:22:59.118210 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:22:59.119037 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:22:59.119037 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:22:59.119037 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:22:59.119037 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:22:59.119037 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 04:22:59.119037 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 04:22:59.119037 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 04:22:59.119037 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 04:22:59.119037 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 04:22:59.119037 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 04:22:59.119037 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 25 04:22:59.119037 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:22:59.121816 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 283/557] 10.222.0.25 () {56 vars in 1102 bytes} [Sun Sep 25 04:22:58 2022] GET /identity/v3/roles/service => generated 85 bytes in 253 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 04:22:59.122799 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8fd2bb1-453a-45ba-8e08-cab54b6b5a32 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:59.145027 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:59.216845 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a8c5190-b0af-4c4a-82c0-cd4b410ff1a7 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:59.245643 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:59.257171 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30b4cb0a-de0c-4781-b935-193d6bb78cb1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:59.257839 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30b4cb0a-de0c-4781-b935-193d6bb78cb1 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:59.258122 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30b4cb0a-de0c-4781-b935-193d6bb78cb1 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:59.258397 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30b4cb0a-de0c-4781-b935-193d6bb78cb1 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:59.267351 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30b4cb0a-de0c-4781-b935-193d6bb78cb1 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:59.268364 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8fd2bb1-453a-45ba-8e08-cab54b6b5a32 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:59.268925 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30b4cb0a-de0c-4781-b935-193d6bb78cb1 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:59.281519 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-30b4cb0a-de0c-4781-b935-193d6bb78cb1 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 04:22:59.281519 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:22:59.281519 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:22:59.281519 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:22:59.281519 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:22:59.281519 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:22:59.281519 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:22:59.281519 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:22:59.281519 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:22:59.281519 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:22:59.281519 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:22:59.281519 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:22:59.281519 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 25 04:22:59.281519 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 25 04:22:59.281519 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 25 04:22:59.282918 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 25 04:22:59.282918 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:22:59.282918 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:22:59.282918 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:22:59.282918 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:22:59.282918 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:22:59.282918 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:22:59.282918 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:22:59.282918 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:22:59.282918 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:22:59.282918 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:22:59.282918 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:22:59.282918 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:22:59.282918 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:22:59.282918 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:22:59.282918 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:22:59.282918 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:22:59.283816 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 25 04:22:59.283816 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 25 04:22:59.283816 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 25 04:22:59.283816 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 25 04:22:59.283816 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 25 04:22:59.283816 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 25 04:22:59.283816 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 04:22:59.283816 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:22:59.283816 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 275/558] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Sep 25 04:22:59 2022] GET /identity/v3/projects/service => generated 88 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:22:59.283816 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30b4cb0a-de0c-4781-b935-193d6bb78cb1 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:59.319198 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-281d0e3c-bb3d-442a-b691-32238bd1c8c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:59.319838 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-281d0e3c-bb3d-442a-b691-32238bd1c8c6 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:59.320117 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-281d0e3c-bb3d-442a-b691-32238bd1c8c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:59.320389 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-281d0e3c-bb3d-442a-b691-32238bd1c8c6 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:59.333067 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-281d0e3c-bb3d-442a-b691-32238bd1c8c6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 04:22:59.338650 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-281d0e3c-bb3d-442a-b691-32238bd1c8c6 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 25 04:22:59.338650 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:22:59.338650 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:22:59.338650 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:22:59.338650 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:22:59.338650 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:22:59.338650 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:22:59.338650 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:22:59.338650 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:22:59.338650 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:22:59.338650 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:22:59.338650 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:22:59.338650 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 04:22:59.338650 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 04:22:59.338650 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 04:22:59.338650 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 04:22:59.339695 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 04:22:59.339695 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 04:22:59.339695 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 04:22:59.339695 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 04:22:59.339695 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:22:59.339695 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:22:59.339695 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:22:59.339695 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:22:59.339695 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:22:59.339695 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:22:59.339695 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:22:59.339695 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:22:59.339695 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:22:59.339695 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:22:59.339695 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:22:59.339695 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:22:59.341238 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:22:59.341238 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:22:59.341238 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:22:59.341238 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:22:59.341238 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 04:22:59.341238 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 04:22:59.341238 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 04:22:59.341238 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 04:22:59.341238 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 04:22:59.341238 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 04:22:59.341238 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 25 04:22:59.341238 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:22:59.345071 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 284/559] 10.222.0.25 () {56 vars in 1102 bytes} [Sun Sep 25 04:22:59 2022] GET /identity/v3/roles/service => generated 85 bytes in 217 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:22:59.346516 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-281d0e3c-bb3d-442a-b691-32238bd1c8c6 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:59.411049 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30b4cb0a-de0c-4781-b935-193d6bb78cb1 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:59.432856 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:59.447440 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac74355b-c6b7-45e4-ac49-aec36bbc650f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:59.448054 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac74355b-c6b7-45e4-ac49-aec36bbc650f None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:59.448321 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac74355b-c6b7-45e4-ac49-aec36bbc650f None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:59.448590 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac74355b-c6b7-45e4-ac49-aec36bbc650f None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:59.449273 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac74355b-c6b7-45e4-ac49-aec36bbc650f None admin] RBAC: Adding query filter params (name=service) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:59.449569 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac74355b-c6b7-45e4-ac49-aec36bbc650f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:59.451187 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac74355b-c6b7-45e4-ac49-aec36bbc650f None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:59.460669 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 276/560] 10.222.0.25 () {56 vars in 1103 bytes} [Sun Sep 25 04:22:59 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:59.460816 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-281d0e3c-bb3d-442a-b691-32238bd1c8c6 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:59.465510 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac74355b-c6b7-45e4-ac49-aec36bbc650f None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:59.496181 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78adea7c-3240-4eb3-b3f4-53093623ceba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:59.496822 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78adea7c-3240-4eb3-b3f4-53093623ceba None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:59.497107 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78adea7c-3240-4eb3-b3f4-53093623ceba None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:59.497373 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78adea7c-3240-4eb3-b3f4-53093623ceba None admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:59.498059 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78adea7c-3240-4eb3-b3f4-53093623ceba None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:59.498366 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78adea7c-3240-4eb3-b3f4-53093623ceba None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:59.500321 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78adea7c-3240-4eb3-b3f4-53093623ceba None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:59.512982 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 285/561] 10.222.0.25 () {56 vars in 1148 bytes} [Sun Sep 25 04:22:59 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:59.514174 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20a95206-e27a-447a-bd35-83fb97d8d2ea None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:59.514457 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20a95206-e27a-447a-bd35-83fb97d8d2ea None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:59.514724 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20a95206-e27a-447a-bd35-83fb97d8d2ea None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:59.516883 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 286/562] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:22:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:22:59.518652 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:59.521915 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f69df8f-8b99-4aa4-a34f-b1c1b6a90be9 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:59.522196 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f69df8f-8b99-4aa4-a34f-b1c1b6a90be9 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:59.522466 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f69df8f-8b99-4aa4-a34f-b1c1b6a90be9 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:59.523699 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 287/563] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:22:59 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:22:59.524380 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f69df8f-8b99-4aa4-a34f-b1c1b6a90be9 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:59.570073 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac74355b-c6b7-45e4-ac49-aec36bbc650f None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:59.604150 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-635c3d98-305f-46f5-bdd9-905441f7fa71 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:59.604785 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-635c3d98-305f-46f5-bdd9-905441f7fa71 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:59.605067 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-635c3d98-305f-46f5-bdd9-905441f7fa71 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:59.605339 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-635c3d98-305f-46f5-bdd9-905441f7fa71 None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:59.605991 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-635c3d98-305f-46f5-bdd9-905441f7fa71 None admin] RBAC: Adding query filter params (name=service) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:59.606291 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-635c3d98-305f-46f5-bdd9-905441f7fa71 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:59.607639 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-635c3d98-305f-46f5-bdd9-905441f7fa71 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:59.615792 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 277/564] 10.222.0.25 () {56 vars in 1103 bytes} [Sun Sep 25 04:22:59 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:59.616575 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-635c3d98-305f-46f5-bdd9-905441f7fa71 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:59.630625 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f69df8f-8b99-4aa4-a34f-b1c1b6a90be9 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:59.663468 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b57e61c-3458-4ea8-a837-3a50b610ce7f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:59.664097 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b57e61c-3458-4ea8-a837-3a50b610ce7f None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:59.664378 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b57e61c-3458-4ea8-a837-3a50b610ce7f None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:59.664672 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b57e61c-3458-4ea8-a837-3a50b610ce7f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:59.667634 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:59.676292 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b57e61c-3458-4ea8-a837-3a50b610ce7f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:59.677777 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b57e61c-3458-4ea8-a837-3a50b610ce7f None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:59.693130 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-7b57e61c-3458-4ea8-a837-3a50b610ce7f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:22:59.693130 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:22:59.693130 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:22:59.693130 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:22:59.693130 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:22:59.693130 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:22:59.693130 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:22:59.693130 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:22:59.693130 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:22:59.693130 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:22:59.693130 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:22:59.693130 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:22:59.693130 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 04:22:59.693130 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 04:22:59.693130 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 04:22:59.693130 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 04:22:59.694104 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:22:59.694104 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:22:59.694104 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:22:59.694104 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:22:59.694104 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:22:59.694104 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:22:59.694104 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:22:59.694104 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:22:59.694104 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:22:59.694104 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:22:59.694104 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:22:59.694104 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:22:59.694104 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:22:59.694104 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:22:59.694104 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:22:59.694104 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:22:59.694104 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 04:22:59.695088 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 04:22:59.695088 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:22:59.695088 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:22:59.695088 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 288/565] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:22:59 2022] GET /identity/v3/domains/Default => generated 87 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:22:59.695331 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d562f9be-65be-4bcf-97ff-796adb304412 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:59.695590 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d562f9be-65be-4bcf-97ff-796adb304412 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:59.695846 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d562f9be-65be-4bcf-97ff-796adb304412 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:59.713473 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-d562f9be-65be-4bcf-97ff-796adb304412 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:22:59.714683 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-d562f9be-65be-4bcf-97ff-796adb304412 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:22:59.719126 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-635c3d98-305f-46f5-bdd9-905441f7fa71 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:59.729244 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:59.752834 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-479c793b-f94c-44c2-8b7f-5c0f9be1fd96 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:59.753368 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-479c793b-f94c-44c2-8b7f-5c0f9be1fd96 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:59.753652 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-479c793b-f94c-44c2-8b7f-5c0f9be1fd96 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:59.753926 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-479c793b-f94c-44c2-8b7f-5c0f9be1fd96 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:59.754643 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-479c793b-f94c-44c2-8b7f-5c0f9be1fd96 None admin] RBAC: Adding query filter params (role.id=9b44520f482c4423bd10568d0f86718c, scope.project.id=056ca1b1a9974a7581550de0d9217823, user.id=23ea88f46e7740d295148a93ade29dd9) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:22:59.754954 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-479c793b-f94c-44c2-8b7f-5c0f9be1fd96 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:59.756382 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-479c793b-f94c-44c2-8b7f-5c0f9be1fd96 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:22:59.761711 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 278/566] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Sep 25 04:22:59 2022] GET /identity/v3/role_assignments?user.id=23ea88f46e7740d295148a93ade29dd9&scope.project.id=056ca1b1a9974a7581550de0d9217823&role.id=9b44520f482c4423bd10568d0f86718c => generated 259 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:22:59.762922 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-713751ba-16c9-45f7-96b4-d31b6cf7c913 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:59.763212 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-713751ba-16c9-45f7-96b4-d31b6cf7c913 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:59.763487 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-713751ba-16c9-45f7-96b4-d31b6cf7c913 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:59.800836 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 289/567] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:22:59 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 04:22:59.800836 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d562f9be-65be-4bcf-97ff-796adb304412 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:59.803157 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-713751ba-16c9-45f7-96b4-d31b6cf7c913 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:22:59.805572 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-713751ba-16c9-45f7-96b4-d31b6cf7c913 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:22:59.904178 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 279/568] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:22:59 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 142 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 04:22:59.905686 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-713751ba-16c9-45f7-96b4-d31b6cf7c913 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:22:59.912122 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d562f9be-65be-4bcf-97ff-796adb304412 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:22:59.935019 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:22:59.969478 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6609d2c-0aae-4bf7-97ef-fbc7f3a13190 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:22:59.970123 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6609d2c-0aae-4bf7-97ef-fbc7f3a13190 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:22:59.970397 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6609d2c-0aae-4bf7-97ef-fbc7f3a13190 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:22:59.970683 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6609d2c-0aae-4bf7-97ef-fbc7f3a13190 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:22:59.985147 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6609d2c-0aae-4bf7-97ef-fbc7f3a13190 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:22:59.987073 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6609d2c-0aae-4bf7-97ef-fbc7f3a13190 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:00.008439 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-f6609d2c-0aae-4bf7-97ef-fbc7f3a13190 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:23:00.008439 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:00.008439 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:00.008439 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:00.008439 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:00.008439 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:00.008439 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:00.008439 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:00.008439 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:00.008439 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:00.008439 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:00.008439 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:00.008439 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 04:23:00.008439 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 04:23:00.008439 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 04:23:00.008439 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 04:23:00.009472 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:00.009472 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:00.009472 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:00.009472 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:00.009472 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:00.009472 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:00.009472 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:00.009472 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:00.009472 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:00.009472 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:00.009472 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:00.009472 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:00.009472 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:00.009472 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:00.009472 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:00.009472 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:00.009472 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 04:23:00.010551 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 04:23:00.010551 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:23:00.010551 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:23:00.010551 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 290/569] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:22:59 2022] GET /identity/v3/domains/Default => generated 87 bytes in 222 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:00.016671 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6609d2c-0aae-4bf7-97ef-fbc7f3a13190 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:00.088352 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-713751ba-16c9-45f7-96b4-d31b6cf7c913 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:00.091664 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:00.129461 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edf91997-a64f-416a-8ec1-daf86d5f8374 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:00.130377 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edf91997-a64f-416a-8ec1-daf86d5f8374 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:00.130844 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edf91997-a64f-416a-8ec1-daf86d5f8374 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:00.131328 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edf91997-a64f-416a-8ec1-daf86d5f8374 None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:00.132256 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edf91997-a64f-416a-8ec1-daf86d5f8374 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:00.132887 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edf91997-a64f-416a-8ec1-daf86d5f8374 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:00.134481 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edf91997-a64f-416a-8ec1-daf86d5f8374 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:00.151461 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6609d2c-0aae-4bf7-97ef-fbc7f3a13190 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:00.152151 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 280/570] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Sep 25 04:22:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 242 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:00.153306 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edf91997-a64f-416a-8ec1-daf86d5f8374 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:00.191574 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcb1e869-7fd9-49a1-ad4a-8e6e8e5f9d71 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:00.192186 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcb1e869-7fd9-49a1-ad4a-8e6e8e5f9d71 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:00.192465 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcb1e869-7fd9-49a1-ad4a-8e6e8e5f9d71 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:00.192768 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcb1e869-7fd9-49a1-ad4a-8e6e8e5f9d71 None admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:00.193582 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcb1e869-7fd9-49a1-ad4a-8e6e8e5f9d71 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:00.193949 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcb1e869-7fd9-49a1-ad4a-8e6e8e5f9d71 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:00.196155 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcb1e869-7fd9-49a1-ad4a-8e6e8e5f9d71 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:00.209854 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 291/571] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Sep 25 04:23:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:00.210689 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcb1e869-7fd9-49a1-ad4a-8e6e8e5f9d71 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:00.247060 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:00.274834 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edf91997-a64f-416a-8ec1-daf86d5f8374 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:00.311712 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a86f53f-8163-443b-a5f4-f4a8fa8f96dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:00.313290 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a86f53f-8163-443b-a5f4-f4a8fa8f96dd None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:00.313585 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a86f53f-8163-443b-a5f4-f4a8fa8f96dd None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:00.314013 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a86f53f-8163-443b-a5f4-f4a8fa8f96dd None admin] PATH_INFO: `/v3/roles/service` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:00.320240 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcb1e869-7fd9-49a1-ad4a-8e6e8e5f9d71 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:00.325247 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a86f53f-8163-443b-a5f4-f4a8fa8f96dd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 04:23:00.331583 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-6a86f53f-8163-443b-a5f4-f4a8fa8f96dd None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 25 04:23:00.331583 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:00.331583 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:00.331583 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:00.331583 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:00.331583 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:00.331583 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:00.331583 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:00.331583 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:00.331583 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:00.331583 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:00.331583 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:00.331583 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 04:23:00.331583 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 04:23:00.331583 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 04:23:00.331583 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 04:23:00.332773 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 04:23:00.332773 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 04:23:00.332773 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 04:23:00.332773 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 04:23:00.332773 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:00.332773 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:00.332773 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:00.332773 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:00.332773 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:00.332773 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:00.332773 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:00.332773 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:00.332773 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:00.332773 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:00.332773 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:00.332773 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:00.333900 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:00.333900 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:00.333900 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:00.333900 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:00.333900 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 04:23:00.333900 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 04:23:00.333900 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 04:23:00.333900 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 04:23:00.333900 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 04:23:00.333900 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 04:23:00.333900 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 25 04:23:00.333900 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:00.333900 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 281/572] 10.222.0.25 () {56 vars in 1102 bytes} [Sun Sep 25 04:23:00 2022] GET /identity/v3/roles/service => generated 85 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:00.333900 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a86f53f-8163-443b-a5f4-f4a8fa8f96dd None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:00.361045 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:00.363716 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77b190b1-5610-420a-a547-534ee1ff1630 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:00.364361 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b190b1-5610-420a-a547-534ee1ff1630 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:00.364723 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b190b1-5610-420a-a547-534ee1ff1630 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:00.365031 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b190b1-5610-420a-a547-534ee1ff1630 None admin] PATH_INFO: `/v3/users/glance` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:00.372421 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77b190b1-5610-420a-a547-534ee1ff1630 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:00.373918 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77b190b1-5610-420a-a547-534ee1ff1630 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:00.387082 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-77b190b1-5610-420a-a547-534ee1ff1630 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Sep 25 04:23:00.387082 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:00.387082 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:00.387082 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:00.387082 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:00.387082 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:00.387082 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:00.387082 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:00.387082 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:00.387082 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:00.387082 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:00.387082 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:00.387082 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 25 04:23:00.387082 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 25 04:23:00.387082 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 25 04:23:00.387082 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 25 04:23:00.388124 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:00.388124 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:00.388124 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 25 04:23:00.388124 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 04:23:00.388124 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 25 04:23:00.388124 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 04:23:00.388124 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:00.388124 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:00.388124 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:00.388124 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:00.388124 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:00.388124 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:00.388124 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:00.388124 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:00.388124 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:00.388124 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:00.388124 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:00.388124 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:00.389234 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:00.389234 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:00.389234 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 25 04:23:00.389234 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 25 04:23:00.389234 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 25 04:23:00.389234 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 25 04:23:00.389234 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 25 04:23:00.389234 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 25 04:23:00.389234 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Sep 25 04:23:00.389234 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:23:00.389234 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 292/573] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Sep 25 04:23:00 2022] GET /identity/v3/users/glance => generated 84 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:00.389234 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77b190b1-5610-420a-a547-534ee1ff1630 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:00.449394 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a86f53f-8163-443b-a5f4-f4a8fa8f96dd None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:00.491958 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aaae9d0-bdb6-4730-a926-47147ecab85a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:00.492606 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aaae9d0-bdb6-4730-a926-47147ecab85a None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:00.492920 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aaae9d0-bdb6-4730-a926-47147ecab85a None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:00.493194 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aaae9d0-bdb6-4730-a926-47147ecab85a None admin] PATH_INFO: `/v3/roles/service` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:00.494715 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:00.504596 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aaae9d0-bdb6-4730-a926-47147ecab85a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 04:23:00.506529 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77b190b1-5610-420a-a547-534ee1ff1630 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:00.509847 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-0aaae9d0-bdb6-4730-a926-47147ecab85a None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 25 04:23:00.509847 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:00.509847 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:00.509847 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:00.509847 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:00.509847 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:00.509847 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:00.509847 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:00.509847 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:00.509847 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:00.509847 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:00.509847 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:00.509847 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 04:23:00.509847 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 04:23:00.509847 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 04:23:00.509847 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 04:23:00.510840 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 04:23:00.510840 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 04:23:00.510840 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 04:23:00.510840 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 04:23:00.510840 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:00.510840 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:00.510840 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:00.510840 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:00.510840 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:00.510840 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:00.510840 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:00.510840 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:00.510840 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:00.510840 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:00.510840 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:00.510840 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:00.512272 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:00.512272 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:00.512272 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:00.512272 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:00.512272 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 04:23:00.512272 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 04:23:00.512272 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 04:23:00.512272 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 04:23:00.512272 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 04:23:00.512272 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 04:23:00.512272 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 25 04:23:00.512272 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:00.512272 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 282/574] 10.222.0.25 () {56 vars in 1102 bytes} [Sun Sep 25 04:23:00 2022] GET /identity/v3/roles/service => generated 85 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:00.512272 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aaae9d0-bdb6-4730-a926-47147ecab85a None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:00.546849 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7ec7974-286b-4194-9bb1-397a39de9a91 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:00.547843 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7ec7974-286b-4194-9bb1-397a39de9a91 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:00.548339 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7ec7974-286b-4194-9bb1-397a39de9a91 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:00.548971 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7ec7974-286b-4194-9bb1-397a39de9a91 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:00.561516 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7ec7974-286b-4194-9bb1-397a39de9a91 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:00.563236 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7ec7974-286b-4194-9bb1-397a39de9a91 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:00.574800 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-d7ec7974-286b-4194-9bb1-397a39de9a91 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Sep 25 04:23:00.574800 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:00.574800 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:00.574800 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:00.574800 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:00.574800 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:00.574800 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:00.574800 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:00.574800 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:00.574800 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:00.574800 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:00.574800 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:00.574800 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 25 04:23:00.574800 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 25 04:23:00.574800 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 25 04:23:00.574800 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 25 04:23:00.576042 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:00.576042 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:00.576042 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 25 04:23:00.576042 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 04:23:00.576042 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 25 04:23:00.576042 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 04:23:00.576042 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:00.576042 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:00.576042 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:00.576042 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:00.576042 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:00.576042 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:00.576042 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:00.576042 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:00.576042 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:00.576042 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:00.576042 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:00.576042 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:00.577518 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:00.577518 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:00.577518 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 25 04:23:00.577518 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 25 04:23:00.577518 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 25 04:23:00.577518 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 25 04:23:00.577518 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 25 04:23:00.577518 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 25 04:23:00.577518 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Sep 25 04:23:00.577518 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:23:00.577518 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 293/575] 10.222.0.25 () {56 vars in 1102 bytes} [Sun Sep 25 04:23:00 2022] GET /identity/v3/users/neutron => generated 85 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:00.577518 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7ec7974-286b-4194-9bb1-397a39de9a91 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:00.607006 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:00.624262 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aaae9d0-bdb6-4730-a926-47147ecab85a None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:00.658776 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4a63f11-78fb-4a70-8303-e8c03c570d51 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:00.659388 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a63f11-78fb-4a70-8303-e8c03c570d51 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:00.659670 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a63f11-78fb-4a70-8303-e8c03c570d51 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:00.659944 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a63f11-78fb-4a70-8303-e8c03c570d51 None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:00.660821 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4a63f11-78fb-4a70-8303-e8c03c570d51 None admin] RBAC: Adding query filter params (name=service) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:00.661129 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4a63f11-78fb-4a70-8303-e8c03c570d51 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:00.662630 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4a63f11-78fb-4a70-8303-e8c03c570d51 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:00.676383 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 283/576] 10.222.0.25 () {56 vars in 1103 bytes} [Sun Sep 25 04:23:00 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:00.676383 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4a63f11-78fb-4a70-8303-e8c03c570d51 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:00.689380 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7ec7974-286b-4194-9bb1-397a39de9a91 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:00.726552 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24216c6b-a90b-4643-9752-fd0995ce5d62 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:00.727299 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24216c6b-a90b-4643-9752-fd0995ce5d62 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:00.727680 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24216c6b-a90b-4643-9752-fd0995ce5d62 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:00.728013 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24216c6b-a90b-4643-9752-fd0995ce5d62 None admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:00.728864 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24216c6b-a90b-4643-9752-fd0995ce5d62 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:00.729174 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24216c6b-a90b-4643-9752-fd0995ce5d62 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:00.730615 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24216c6b-a90b-4643-9752-fd0995ce5d62 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:00.772840 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 294/577] 10.222.0.25 () {56 vars in 1137 bytes} [Sun Sep 25 04:23:00 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:00.772840 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24216c6b-a90b-4643-9752-fd0995ce5d62 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:00.781947 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:00.789087 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4a63f11-78fb-4a70-8303-e8c03c570d51 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:00.824135 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e71101b9-ad68-4805-ae7f-1bbfa03bcc2a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:00.824963 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e71101b9-ad68-4805-ae7f-1bbfa03bcc2a None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:00.825244 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e71101b9-ad68-4805-ae7f-1bbfa03bcc2a None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:00.825769 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e71101b9-ad68-4805-ae7f-1bbfa03bcc2a None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:00.826421 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e71101b9-ad68-4805-ae7f-1bbfa03bcc2a None admin] RBAC: Adding query filter params (name=service) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:00.826722 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e71101b9-ad68-4805-ae7f-1bbfa03bcc2a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:00.829669 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e71101b9-ad68-4805-ae7f-1bbfa03bcc2a None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:00.844101 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 284/578] 10.222.0.25 () {56 vars in 1103 bytes} [Sun Sep 25 04:23:00 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:00.844101 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e71101b9-ad68-4805-ae7f-1bbfa03bcc2a None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:00.857390 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:00.876369 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24216c6b-a90b-4643-9752-fd0995ce5d62 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:00.911859 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a460df9-8e89-4fb0-8940-c1f60524a824 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:00.912476 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a460df9-8e89-4fb0-8940-c1f60524a824 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:00.912825 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a460df9-8e89-4fb0-8940-c1f60524a824 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:00.913104 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a460df9-8e89-4fb0-8940-c1f60524a824 None admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:00.913995 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a460df9-8e89-4fb0-8940-c1f60524a824 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:00.914313 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a460df9-8e89-4fb0-8940-c1f60524a824 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:00.916086 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a460df9-8e89-4fb0-8940-c1f60524a824 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:00.938075 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 295/579] 10.222.0.25 () {56 vars in 1139 bytes} [Sun Sep 25 04:23:00 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:00.938075 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a460df9-8e89-4fb0-8940-c1f60524a824 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:00.966552 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e71101b9-ad68-4805-ae7f-1bbfa03bcc2a None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:01.002429 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7be2c379-eeb3-46fe-a783-9f1148758ec0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:01.003055 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7be2c379-eeb3-46fe-a783-9f1148758ec0 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:01.003397 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7be2c379-eeb3-46fe-a783-9f1148758ec0 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:01.003678 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7be2c379-eeb3-46fe-a783-9f1148758ec0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:01.004495 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:01.016331 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7be2c379-eeb3-46fe-a783-9f1148758ec0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:01.017968 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7be2c379-eeb3-46fe-a783-9f1148758ec0 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:01.033573 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-7be2c379-eeb3-46fe-a783-9f1148758ec0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:23:01.033573 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:01.033573 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:01.033573 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:01.033573 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:01.033573 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:01.033573 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:01.033573 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:01.033573 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:01.033573 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:01.033573 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:01.033573 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:01.033573 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 04:23:01.033573 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 04:23:01.033573 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 04:23:01.033573 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 04:23:01.034548 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:01.034548 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:01.034548 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:01.034548 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:01.034548 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:01.034548 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:01.034548 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:01.034548 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:01.034548 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:01.034548 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:01.034548 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:01.034548 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:01.034548 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:01.034548 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:01.034548 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:01.034548 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:01.034548 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 04:23:01.035445 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 04:23:01.035445 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:23:01.035445 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:01.035445 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 285/580] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:23:00 2022] GET /identity/v3/domains/Default => generated 87 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:01.035680 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7be2c379-eeb3-46fe-a783-9f1148758ec0 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:01.045125 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a460df9-8e89-4fb0-8940-c1f60524a824 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:01.081841 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1152d59-75cc-4a02-95a6-365ca78a6256 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:01.082393 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1152d59-75cc-4a02-95a6-365ca78a6256 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:01.082670 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1152d59-75cc-4a02-95a6-365ca78a6256 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:01.082930 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1152d59-75cc-4a02-95a6-365ca78a6256 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:01.092947 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1152d59-75cc-4a02-95a6-365ca78a6256 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:01.095146 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1152d59-75cc-4a02-95a6-365ca78a6256 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:01.100879 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:01.112544 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-e1152d59-75cc-4a02-95a6-365ca78a6256 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:23:01.112544 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:01.112544 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:01.112544 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:01.112544 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:01.112544 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:01.112544 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:01.112544 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:01.112544 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:01.112544 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:01.112544 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:01.112544 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:01.112544 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 04:23:01.112544 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 04:23:01.112544 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 04:23:01.112544 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 04:23:01.113652 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:01.113652 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:01.113652 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:01.113652 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:01.113652 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:01.113652 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:01.113652 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:01.113652 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:01.113652 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:01.113652 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:01.113652 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:01.113652 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:01.113652 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:01.113652 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:01.113652 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:01.113652 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:01.113652 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 04:23:01.114555 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 04:23:01.114555 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:23:01.114555 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:23:01.114555 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 296/581] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:23:00 2022] GET /identity/v3/domains/Default => generated 87 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:01.114555 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1152d59-75cc-4a02-95a6-365ca78a6256 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:01.154460 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7be2c379-eeb3-46fe-a783-9f1148758ec0 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:01.190033 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48158a3a-c2e0-42e7-b7e9-71156b20d33c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:01.190662 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48158a3a-c2e0-42e7-b7e9-71156b20d33c None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:01.190938 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48158a3a-c2e0-42e7-b7e9-71156b20d33c None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:01.191234 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48158a3a-c2e0-42e7-b7e9-71156b20d33c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:01.202225 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48158a3a-c2e0-42e7-b7e9-71156b20d33c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:01.204035 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48158a3a-c2e0-42e7-b7e9-71156b20d33c None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:01.218063 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-48158a3a-c2e0-42e7-b7e9-71156b20d33c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:23:01.218063 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:01.218063 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:01.218063 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:01.218063 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:01.218063 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:01.218063 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:01.218063 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:01.218063 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:01.218063 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:01.218063 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:01.218063 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:01.218063 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 04:23:01.218063 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 04:23:01.218063 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 04:23:01.218063 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 04:23:01.219554 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:01.219554 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:01.219554 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:01.219554 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:01.219554 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:01.219554 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:01.219554 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:01.219554 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:01.219554 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:01.219554 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:01.219554 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:01.219554 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:01.219554 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:01.219554 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:01.219554 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:01.219554 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:01.219554 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 04:23:01.220479 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 04:23:01.220479 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:23:01.220479 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:01.220479 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 286/582] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:23:01 2022] GET /identity/v3/domains/Default => generated 87 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:01.221245 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48158a3a-c2e0-42e7-b7e9-71156b20d33c None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:01.228228 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1152d59-75cc-4a02-95a6-365ca78a6256 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:01.245858 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:01.264529 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52a2cdd3-7bb3-40eb-95fe-0e5cf4c600bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:01.265205 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52a2cdd3-7bb3-40eb-95fe-0e5cf4c600bf None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:01.265624 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52a2cdd3-7bb3-40eb-95fe-0e5cf4c600bf None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:01.265901 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52a2cdd3-7bb3-40eb-95fe-0e5cf4c600bf None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:01.276687 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52a2cdd3-7bb3-40eb-95fe-0e5cf4c600bf None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:01.278133 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52a2cdd3-7bb3-40eb-95fe-0e5cf4c600bf None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:01.292925 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-52a2cdd3-7bb3-40eb-95fe-0e5cf4c600bf None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:23:01.292925 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:01.292925 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:01.292925 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:01.292925 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:01.292925 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:01.292925 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:01.292925 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:01.292925 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:01.292925 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:01.292925 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:01.292925 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:01.292925 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 04:23:01.292925 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 04:23:01.292925 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 04:23:01.292925 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 04:23:01.294048 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:01.294048 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:01.294048 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:01.294048 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:01.294048 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:01.294048 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:01.294048 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:01.294048 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:01.294048 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:01.294048 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:01.294048 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:01.294048 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:01.294048 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:01.294048 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:01.294048 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:01.294048 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:01.294048 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 04:23:01.294980 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 04:23:01.294980 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:23:01.294980 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:23:01.294980 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 297/583] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:23:01 2022] GET /identity/v3/domains/Default => generated 87 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:01.294980 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52a2cdd3-7bb3-40eb-95fe-0e5cf4c600bf None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:01.333000 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48158a3a-c2e0-42e7-b7e9-71156b20d33c None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:01.345872 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:01.373764 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a095709-3ff6-4691-84b7-bfdb774b2697 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:01.374370 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a095709-3ff6-4691-84b7-bfdb774b2697 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:01.375437 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a095709-3ff6-4691-84b7-bfdb774b2697 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:01.375437 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a095709-3ff6-4691-84b7-bfdb774b2697 None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:01.375640 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a095709-3ff6-4691-84b7-bfdb774b2697 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:01.375923 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a095709-3ff6-4691-84b7-bfdb774b2697 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:01.377320 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a095709-3ff6-4691-84b7-bfdb774b2697 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:01.389308 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 287/584] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Sep 25 04:23:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:01.389957 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a095709-3ff6-4691-84b7-bfdb774b2697 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:01.411609 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52a2cdd3-7bb3-40eb-95fe-0e5cf4c600bf None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:01.446950 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc434597-1988-4abc-bc40-85f882ce861a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:01.447559 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc434597-1988-4abc-bc40-85f882ce861a None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:01.447835 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc434597-1988-4abc-bc40-85f882ce861a None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:01.448109 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc434597-1988-4abc-bc40-85f882ce861a None admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:01.448779 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc434597-1988-4abc-bc40-85f882ce861a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:01.449079 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc434597-1988-4abc-bc40-85f882ce861a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:01.450708 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc434597-1988-4abc-bc40-85f882ce861a None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:01.464481 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 298/585] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Sep 25 04:23:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:01.465973 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc434597-1988-4abc-bc40-85f882ce861a None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:01.502222 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:01.506171 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a095709-3ff6-4691-84b7-bfdb774b2697 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:01.542878 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db70c3b7-f8d4-4805-813e-4d79971f18ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:01.543481 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db70c3b7-f8d4-4805-813e-4d79971f18ce None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:01.543766 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db70c3b7-f8d4-4805-813e-4d79971f18ce None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:01.544037 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db70c3b7-f8d4-4805-813e-4d79971f18ce None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:01.545344 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db70c3b7-f8d4-4805-813e-4d79971f18ce None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:01.545344 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db70c3b7-f8d4-4805-813e-4d79971f18ce None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:01.546325 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db70c3b7-f8d4-4805-813e-4d79971f18ce None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:01.563706 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 288/586] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Sep 25 04:23:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:01.563706 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db70c3b7-f8d4-4805-813e-4d79971f18ce None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:01.585871 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc434597-1988-4abc-bc40-85f882ce861a None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:01.595951 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:01.623670 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3537670f-a23a-4c12-b06a-5c699e15ea58 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:01.624287 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3537670f-a23a-4c12-b06a-5c699e15ea58 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:01.624559 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3537670f-a23a-4c12-b06a-5c699e15ea58 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:01.624893 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3537670f-a23a-4c12-b06a-5c699e15ea58 None admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:01.625875 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3537670f-a23a-4c12-b06a-5c699e15ea58 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:01.626180 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3537670f-a23a-4c12-b06a-5c699e15ea58 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:01.627658 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3537670f-a23a-4c12-b06a-5c699e15ea58 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:01.644672 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 299/587] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Sep 25 04:23:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:01.644672 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3537670f-a23a-4c12-b06a-5c699e15ea58 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:01.669787 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db70c3b7-f8d4-4805-813e-4d79971f18ce None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:01.706263 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16449dcf-63cc-4908-aa4b-ba80e329174d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:01.707277 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16449dcf-63cc-4908-aa4b-ba80e329174d None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:01.707795 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16449dcf-63cc-4908-aa4b-ba80e329174d None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:01.708852 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16449dcf-63cc-4908-aa4b-ba80e329174d None admin] PATH_INFO: `/v3/users/nova` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:01.719461 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16449dcf-63cc-4908-aa4b-ba80e329174d None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:01.721702 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16449dcf-63cc-4908-aa4b-ba80e329174d None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:01.733436 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-16449dcf-63cc-4908-aa4b-ba80e329174d None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 25 04:23:01.733436 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:01.733436 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:01.733436 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:01.733436 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:01.733436 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:01.733436 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:01.733436 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:01.733436 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:01.733436 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:01.733436 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:01.733436 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:01.733436 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 25 04:23:01.733436 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 25 04:23:01.733436 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 25 04:23:01.733436 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 25 04:23:01.734995 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:01.734995 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:01.734995 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 25 04:23:01.734995 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 04:23:01.734995 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 25 04:23:01.734995 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 04:23:01.734995 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:01.734995 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:01.734995 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:01.734995 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:01.734995 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:01.734995 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:01.734995 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:01.734995 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:01.734995 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:01.734995 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:01.734995 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:01.734995 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:01.736242 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:01.736242 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:01.736242 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 25 04:23:01.736242 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 25 04:23:01.736242 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 25 04:23:01.736242 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 25 04:23:01.736242 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 25 04:23:01.736242 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 25 04:23:01.736242 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 25 04:23:01.736242 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:01.740960 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 289/588] 10.222.0.25 () {56 vars in 1093 bytes} [Sun Sep 25 04:23:01 2022] GET /identity/v3/users/nova => generated 82 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:01.742309 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16449dcf-63cc-4908-aa4b-ba80e329174d None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:01.751375 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:01.761943 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3537670f-a23a-4c12-b06a-5c699e15ea58 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:01.797837 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8d9db67-784a-4a23-8bcd-37197c153e46 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:01.798562 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8d9db67-784a-4a23-8bcd-37197c153e46 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:01.798924 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8d9db67-784a-4a23-8bcd-37197c153e46 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:01.799263 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8d9db67-784a-4a23-8bcd-37197c153e46 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:01.806054 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8d9db67-784a-4a23-8bcd-37197c153e46 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:01.807561 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8d9db67-784a-4a23-8bcd-37197c153e46 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:01.818751 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-e8d9db67-784a-4a23-8bcd-37197c153e46 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 04:23:01.818751 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:01.818751 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:01.818751 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:01.818751 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:01.818751 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:01.818751 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:01.818751 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:01.818751 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:01.818751 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:01.818751 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:01.818751 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:01.818751 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 25 04:23:01.818751 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 25 04:23:01.818751 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 25 04:23:01.819874 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 25 04:23:01.819874 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:01.819874 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:01.819874 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:01.819874 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:01.819874 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:01.819874 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:01.819874 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:01.819874 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:01.819874 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:01.819874 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:01.819874 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:01.819874 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:01.819874 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:01.819874 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:01.819874 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:01.819874 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:01.821013 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 25 04:23:01.821013 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 25 04:23:01.821013 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 25 04:23:01.821013 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 25 04:23:01.821013 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 25 04:23:01.821013 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 25 04:23:01.821013 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 04:23:01.821013 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:23:01.821013 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 300/589] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Sep 25 04:23:01 2022] GET /identity/v3/projects/service => generated 88 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:01.821013 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8d9db67-784a-4a23-8bcd-37197c153e46 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:01.849413 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:01.859296 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16449dcf-63cc-4908-aa4b-ba80e329174d None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:01.893737 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-505f9034-94ab-4659-9cba-72f9e2a69446 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:01.894404 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-505f9034-94ab-4659-9cba-72f9e2a69446 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:01.894691 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-505f9034-94ab-4659-9cba-72f9e2a69446 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:01.894968 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-505f9034-94ab-4659-9cba-72f9e2a69446 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:01.904498 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-505f9034-94ab-4659-9cba-72f9e2a69446 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:01.906657 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-505f9034-94ab-4659-9cba-72f9e2a69446 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:01.919064 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-505f9034-94ab-4659-9cba-72f9e2a69446 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Sep 25 04:23:01.919064 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:01.919064 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:01.919064 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:01.919064 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:01.919064 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:01.919064 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:01.919064 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:01.919064 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:01.919064 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:01.919064 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:01.919064 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:01.919064 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 25 04:23:01.919064 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 25 04:23:01.919064 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 25 04:23:01.919064 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 25 04:23:01.920119 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:01.920119 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:01.920119 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 25 04:23:01.920119 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 04:23:01.920119 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 25 04:23:01.920119 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 04:23:01.920119 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:01.920119 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:01.920119 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:01.920119 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:01.920119 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:01.920119 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:01.920119 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:01.920119 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:01.920119 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:01.920119 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:01.920119 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:01.920119 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:01.921133 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:01.921133 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:01.921133 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 25 04:23:01.921133 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 25 04:23:01.921133 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 25 04:23:01.921133 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 25 04:23:01.921133 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 25 04:23:01.921133 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 25 04:23:01.921133 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Sep 25 04:23:01.921133 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:01.921133 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 290/590] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Sep 25 04:23:01 2022] GET /identity/v3/users/cinder => generated 84 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:01.921133 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-505f9034-94ab-4659-9cba-72f9e2a69446 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:01.938678 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8d9db67-784a-4a23-8bcd-37197c153e46 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:01.974791 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca6cb6ff-386f-4a19-86d2-ddeed89d81c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:01.975831 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca6cb6ff-386f-4a19-86d2-ddeed89d81c3 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:01.976293 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca6cb6ff-386f-4a19-86d2-ddeed89d81c3 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:01.976806 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca6cb6ff-386f-4a19-86d2-ddeed89d81c3 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:01.984505 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca6cb6ff-386f-4a19-86d2-ddeed89d81c3 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:01.986330 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca6cb6ff-386f-4a19-86d2-ddeed89d81c3 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:01.998248 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-ca6cb6ff-386f-4a19-86d2-ddeed89d81c3 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 04:23:01.998248 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:01.998248 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:01.998248 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:01.998248 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:01.998248 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:01.998248 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:01.998248 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:01.998248 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:01.998248 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:01.998248 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:01.998248 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:01.998248 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 25 04:23:01.998248 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 25 04:23:01.998248 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 25 04:23:01.999356 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 25 04:23:01.999356 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:01.999356 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:01.999356 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:01.999356 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:01.999356 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:01.999356 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:01.999356 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:01.999356 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:01.999356 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:01.999356 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:01.999356 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:01.999356 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:01.999356 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:01.999356 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:01.999356 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:01.999356 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:02.001206 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 25 04:23:02.001206 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 25 04:23:02.001206 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 25 04:23:02.001206 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 25 04:23:02.001206 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 25 04:23:02.001206 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 25 04:23:02.001206 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 04:23:02.001206 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:23:02.003294 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 301/591] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Sep 25 04:23:01 2022] GET /identity/v3/projects/service => generated 88 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:02.004233 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca6cb6ff-386f-4a19-86d2-ddeed89d81c3 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:02.020976 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:02.027452 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-505f9034-94ab-4659-9cba-72f9e2a69446 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:02.060910 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-158a0a9e-28db-45bf-868b-974d4191c511 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:02.061525 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-158a0a9e-28db-45bf-868b-974d4191c511 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:02.061887 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-158a0a9e-28db-45bf-868b-974d4191c511 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:02.062476 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-158a0a9e-28db-45bf-868b-974d4191c511 None admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:02.064904 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-158a0a9e-28db-45bf-868b-974d4191c511 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:02.065415 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-158a0a9e-28db-45bf-868b-974d4191c511 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:02.067364 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-158a0a9e-28db-45bf-868b-974d4191c511 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:02.086017 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 291/592] 10.222.0.25 () {56 vars in 1133 bytes} [Sun Sep 25 04:23:01 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:02.086894 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-158a0a9e-28db-45bf-868b-974d4191c511 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:02.099873 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:02.114733 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca6cb6ff-386f-4a19-86d2-ddeed89d81c3 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:02.147690 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1beb50f-8ed6-4ff1-998f-dc3a16cebf99 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:02.148295 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1beb50f-8ed6-4ff1-998f-dc3a16cebf99 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:02.148576 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1beb50f-8ed6-4ff1-998f-dc3a16cebf99 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:02.148877 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1beb50f-8ed6-4ff1-998f-dc3a16cebf99 None admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:02.149568 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1beb50f-8ed6-4ff1-998f-dc3a16cebf99 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:02.149876 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1beb50f-8ed6-4ff1-998f-dc3a16cebf99 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:02.151353 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1beb50f-8ed6-4ff1-998f-dc3a16cebf99 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:02.163034 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 302/593] 10.222.0.25 () {56 vars in 1148 bytes} [Sun Sep 25 04:23:02 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:02.163648 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1beb50f-8ed6-4ff1-998f-dc3a16cebf99 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:02.187142 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-158a0a9e-28db-45bf-868b-974d4191c511 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:02.222275 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a30dbef9-38ee-43eb-aab1-6463b164bfe0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:02.222881 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a30dbef9-38ee-43eb-aab1-6463b164bfe0 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:02.223157 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a30dbef9-38ee-43eb-aab1-6463b164bfe0 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:02.223430 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a30dbef9-38ee-43eb-aab1-6463b164bfe0 None admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:02.224148 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a30dbef9-38ee-43eb-aab1-6463b164bfe0 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:02.224446 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a30dbef9-38ee-43eb-aab1-6463b164bfe0 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:02.225892 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a30dbef9-38ee-43eb-aab1-6463b164bfe0 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:02.235988 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:02.242995 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 292/594] 10.222.0.25 () {56 vars in 1137 bytes} [Sun Sep 25 04:23:02 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:02.244201 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2631729a-1feb-4670-b232-bbe112351062 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:02.244485 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2631729a-1feb-4670-b232-bbe112351062 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:02.244792 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2631729a-1feb-4670-b232-bbe112351062 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:02.248679 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 293/595] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:23:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:23:02.248679 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2631729a-1feb-4670-b232-bbe112351062 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:02.276013 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1beb50f-8ed6-4ff1-998f-dc3a16cebf99 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:02.309699 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e56b1266-b730-44ff-9551-c08af16bb09f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:02.310306 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e56b1266-b730-44ff-9551-c08af16bb09f None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:02.310589 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e56b1266-b730-44ff-9551-c08af16bb09f None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:02.310864 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e56b1266-b730-44ff-9551-c08af16bb09f None admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:02.311556 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e56b1266-b730-44ff-9551-c08af16bb09f None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:02.311869 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e56b1266-b730-44ff-9551-c08af16bb09f None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:02.314006 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e56b1266-b730-44ff-9551-c08af16bb09f None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:02.314143 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:02.325730 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 303/596] 10.222.0.25 () {56 vars in 1148 bytes} [Sun Sep 25 04:23:02 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:02.326502 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e56b1266-b730-44ff-9551-c08af16bb09f None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:02.353905 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2631729a-1feb-4670-b232-bbe112351062 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:02.387317 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f134815a-b4d0-4897-89fe-afb0cace9562 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:02.387947 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f134815a-b4d0-4897-89fe-afb0cace9562 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:02.388234 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f134815a-b4d0-4897-89fe-afb0cace9562 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:02.388505 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f134815a-b4d0-4897-89fe-afb0cace9562 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:02.399187 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f134815a-b4d0-4897-89fe-afb0cace9562 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:02.400647 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f134815a-b4d0-4897-89fe-afb0cace9562 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:02.415639 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-f134815a-b4d0-4897-89fe-afb0cace9562 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:23:02.415639 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:02.415639 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:02.415639 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:02.415639 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:02.415639 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:02.415639 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:02.415639 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:02.415639 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:02.415639 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:02.415639 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:02.415639 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:02.415639 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 04:23:02.415639 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 04:23:02.415639 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 04:23:02.415639 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 04:23:02.416915 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:02.416915 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:02.416915 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:02.416915 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:02.416915 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:02.416915 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:02.416915 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:02.416915 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:02.416915 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:02.416915 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:02.416915 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:02.416915 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:02.416915 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:02.416915 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:02.416915 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:02.416915 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:02.416915 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 04:23:02.418069 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 04:23:02.418069 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:23:02.418069 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:02.418069 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 294/597] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:23:02 2022] GET /identity/v3/domains/Default => generated 87 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:02.418069 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f134815a-b4d0-4897-89fe-afb0cace9562 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:02.428381 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e56b1266-b730-44ff-9551-c08af16bb09f None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:02.456704 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:02.463125 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3b964ea-05d8-44ad-901c-14a3bd8c6849 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:02.463787 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3b964ea-05d8-44ad-901c-14a3bd8c6849 None admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:02.464073 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3b964ea-05d8-44ad-901c-14a3bd8c6849 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:02.464343 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3b964ea-05d8-44ad-901c-14a3bd8c6849 None admin] PATH_INFO: `/v3/projects/056ca1b1a9974a7581550de0d9217823/users/316eaa67d5464091af47efd489b96b87/roles/9b44520f482c4423bd10568d0f86718c` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:02.480233 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3b964ea-05d8-44ad-901c-14a3bd8c6849 None admin] RBAC: Authorizing `identity:create_grant(project_id=056ca1b1a9974a7581550de0d9217823, user_id=316eaa67d5464091af47efd489b96b87, role_id=9b44520f482c4423bd10568d0f86718c)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:02.481364 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3b964ea-05d8-44ad-901c-14a3bd8c6849 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:02.495177 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 304/598] 10.222.0.25 () {58 vars in 1439 bytes} [Sun Sep 25 04:23:02 2022] PUT /identity/v3/projects/056ca1b1a9974a7581550de0d9217823/users/316eaa67d5464091af47efd489b96b87/roles/9b44520f482c4423bd10568d0f86718c => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:23:02.496486 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10e920ca-5454-4784-bbaa-59cf4395e5aa None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:02.497014 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10e920ca-5454-4784-bbaa-59cf4395e5aa None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:02.497274 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10e920ca-5454-4784-bbaa-59cf4395e5aa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:02.517521 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-10e920ca-5454-4784-bbaa-59cf4395e5aa None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:23:02.518769 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-10e920ca-5454-4784-bbaa-59cf4395e5aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:23:02.524890 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f134815a-b4d0-4897-89fe-afb0cace9562 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:02.552093 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:02.568577 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07ad8ab1-b461-44f3-8251-a983332ec958 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:02.569262 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ad8ab1-b461-44f3-8251-a983332ec958 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:02.569543 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ad8ab1-b461-44f3-8251-a983332ec958 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:02.569843 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ad8ab1-b461-44f3-8251-a983332ec958 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:02.583712 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07ad8ab1-b461-44f3-8251-a983332ec958 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:02.585381 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07ad8ab1-b461-44f3-8251-a983332ec958 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:02.600559 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-07ad8ab1-b461-44f3-8251-a983332ec958 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:23:02.600559 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:02.600559 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:02.600559 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:02.600559 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:02.600559 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:02.600559 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:02.600559 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:02.600559 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:02.600559 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:02.600559 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:02.600559 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:02.600559 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 04:23:02.600559 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 04:23:02.600559 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 04:23:02.600559 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 04:23:02.601916 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:02.601916 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:02.601916 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:02.601916 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:02.601916 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:02.601916 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:02.601916 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:02.601916 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:02.601916 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:02.601916 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:02.601916 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:02.601916 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:02.601916 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:02.601916 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:02.601916 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:02.601916 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:02.601916 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 04:23:02.602857 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 04:23:02.602857 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:23:02.602857 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:02.602857 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 295/599] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:23:02 2022] GET /identity/v3/domains/Default => generated 87 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:02.603090 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07ad8ab1-b461-44f3-8251-a983332ec958 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:02.607007 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 305/600] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:23:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 04:23:02.608238 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10e920ca-5454-4784-bbaa-59cf4395e5aa None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:02.742451 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10e920ca-5454-4784-bbaa-59cf4395e5aa None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:02.743675 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07ad8ab1-b461-44f3-8251-a983332ec958 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:02.754793 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:02.787208 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8679de5d-97ad-448c-aa1f-237b8b980d3e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:02.788168 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8679de5d-97ad-448c-aa1f-237b8b980d3e None admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:02.788606 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8679de5d-97ad-448c-aa1f-237b8b980d3e None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:02.789124 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8679de5d-97ad-448c-aa1f-237b8b980d3e None admin] PATH_INFO: `/v3/projects/056ca1b1a9974a7581550de0d9217823/users/0268e88e0c5645a98bab5664be6e49bc/roles/9b44520f482c4423bd10568d0f86718c` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:02.791720 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5835e04-3a4a-4a9a-9b67-ea52112a0c29 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:02.792588 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5835e04-3a4a-4a9a-9b67-ea52112a0c29 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:02.792899 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5835e04-3a4a-4a9a-9b67-ea52112a0c29 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:02.793167 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5835e04-3a4a-4a9a-9b67-ea52112a0c29 None admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:02.794721 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5835e04-3a4a-4a9a-9b67-ea52112a0c29 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:02.795026 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5835e04-3a4a-4a9a-9b67-ea52112a0c29 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:02.796384 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5835e04-3a4a-4a9a-9b67-ea52112a0c29 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:02.808249 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8679de5d-97ad-448c-aa1f-237b8b980d3e None admin] RBAC: Authorizing `identity:create_grant(project_id=056ca1b1a9974a7581550de0d9217823, user_id=0268e88e0c5645a98bab5664be6e49bc, role_id=9b44520f482c4423bd10568d0f86718c)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:02.810010 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8679de5d-97ad-448c-aa1f-237b8b980d3e None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:02.812720 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 306/601] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Sep 25 04:23:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:02.813813 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5835e04-3a4a-4a9a-9b67-ea52112a0c29 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:02.831178 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 296/602] 10.222.0.25 () {58 vars in 1439 bytes} [Sun Sep 25 04:23:02 2022] PUT /identity/v3/projects/056ca1b1a9974a7581550de0d9217823/users/0268e88e0c5645a98bab5664be6e49bc/roles/9b44520f482c4423bd10568d0f86718c => generated 0 bytes in 229 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:23:02.832296 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8679de5d-97ad-448c-aa1f-237b8b980d3e None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:02.873585 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:02.964507 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5835e04-3a4a-4a9a-9b67-ea52112a0c29 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:02.990373 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8679de5d-97ad-448c-aa1f-237b8b980d3e None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:03.003196 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ed897ba-a78b-4a31-b4f6-19e23c9c721b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:03.004367 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ed897ba-a78b-4a31-b4f6-19e23c9c721b None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:03.004851 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ed897ba-a78b-4a31-b4f6-19e23c9c721b None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:03.005130 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ed897ba-a78b-4a31-b4f6-19e23c9c721b None admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:03.005786 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ed897ba-a78b-4a31-b4f6-19e23c9c721b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:03.006249 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ed897ba-a78b-4a31-b4f6-19e23c9c721b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:03.007745 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ed897ba-a78b-4a31-b4f6-19e23c9c721b None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:03.022405 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 307/603] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Sep 25 04:23:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:03.023295 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ed897ba-a78b-4a31-b4f6-19e23c9c721b None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:03.036134 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26be2080-7826-49ff-bb9a-c8bc4130ebf3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:03.036820 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26be2080-7826-49ff-bb9a-c8bc4130ebf3 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:03.037161 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26be2080-7826-49ff-bb9a-c8bc4130ebf3 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:03.037433 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26be2080-7826-49ff-bb9a-c8bc4130ebf3 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:03.045577 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26be2080-7826-49ff-bb9a-c8bc4130ebf3 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:03.047687 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26be2080-7826-49ff-bb9a-c8bc4130ebf3 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:03.058039 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:03.059741 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-26be2080-7826-49ff-bb9a-c8bc4130ebf3 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 04:23:03.059741 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:03.059741 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:03.059741 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:03.059741 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:03.059741 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:03.059741 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:03.059741 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:03.059741 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:03.059741 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:03.059741 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:03.059741 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:03.059741 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 25 04:23:03.059741 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 25 04:23:03.059741 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 25 04:23:03.060827 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 25 04:23:03.060827 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:03.060827 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:03.060827 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:03.060827 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:03.060827 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:03.060827 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:03.060827 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:03.060827 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:03.060827 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:03.060827 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:03.060827 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:03.060827 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:03.060827 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:03.060827 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:03.060827 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:03.060827 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:03.061879 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 25 04:23:03.061879 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 25 04:23:03.061879 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 25 04:23:03.061879 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 25 04:23:03.061879 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 25 04:23:03.061879 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 25 04:23:03.061879 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 04:23:03.061879 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:03.061879 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 297/604] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Sep 25 04:23:02 2022] GET /identity/v3/projects/service => generated 88 bytes in 229 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:03.064499 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26be2080-7826-49ff-bb9a-c8bc4130ebf3 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:03.148256 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ed897ba-a78b-4a31-b4f6-19e23c9c721b None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:03.162458 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:03.189464 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26be2080-7826-49ff-bb9a-c8bc4130ebf3 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:03.194941 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88adacc0-c773-417a-8a42-c4a45e307f1a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:03.195887 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88adacc0-c773-417a-8a42-c4a45e307f1a None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:03.196402 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88adacc0-c773-417a-8a42-c4a45e307f1a None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:03.197380 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88adacc0-c773-417a-8a42-c4a45e307f1a None admin] PATH_INFO: `/v3/roles/service` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:03.214666 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88adacc0-c773-417a-8a42-c4a45e307f1a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 04:23:03.220112 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-88adacc0-c773-417a-8a42-c4a45e307f1a None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 25 04:23:03.220112 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:03.220112 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:03.220112 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:03.220112 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:03.220112 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:03.220112 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:03.220112 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:03.220112 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:03.220112 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:03.220112 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:03.220112 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:03.220112 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 04:23:03.220112 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 04:23:03.220112 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 04:23:03.220112 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 04:23:03.221195 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 04:23:03.221195 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 04:23:03.221195 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 04:23:03.221195 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 04:23:03.221195 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:03.221195 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:03.221195 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:03.221195 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:03.221195 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:03.221195 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:03.221195 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:03.221195 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:03.221195 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:03.221195 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:03.221195 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:03.221195 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:03.222649 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:03.222649 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:03.222649 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:03.222649 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:03.222649 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 04:23:03.222649 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 04:23:03.222649 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 04:23:03.222649 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 04:23:03.222649 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 04:23:03.222649 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 04:23:03.222649 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 25 04:23:03.222649 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:23:03.222649 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 308/605] 10.222.0.25 () {56 vars in 1102 bytes} [Sun Sep 25 04:23:03 2022] GET /identity/v3/roles/service => generated 85 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:03.222649 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88adacc0-c773-417a-8a42-c4a45e307f1a None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:03.241414 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8f1c388-46a4-4d69-82c1-3266ab97e1b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:03.242663 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f1c388-46a4-4d69-82c1-3266ab97e1b4 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:03.243190 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f1c388-46a4-4d69-82c1-3266ab97e1b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:03.244122 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f1c388-46a4-4d69-82c1-3266ab97e1b4 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:03.254227 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8f1c388-46a4-4d69-82c1-3266ab97e1b4 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:03.256510 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8f1c388-46a4-4d69-82c1-3266ab97e1b4 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:03.272666 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-a8f1c388-46a4-4d69-82c1-3266ab97e1b4 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 04:23:03.272666 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:03.272666 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:03.272666 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:03.272666 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:03.272666 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:03.272666 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:03.272666 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:03.272666 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:03.272666 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:03.272666 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:03.272666 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:03.272666 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 25 04:23:03.272666 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 25 04:23:03.272666 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 25 04:23:03.273642 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 25 04:23:03.273642 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:03.273642 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:03.273642 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:03.273642 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:03.273642 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:03.273642 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:03.273642 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:03.273642 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:03.273642 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:03.273642 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:03.273642 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:03.273642 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:03.273642 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:03.273642 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:03.273642 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:03.273642 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:03.274557 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 25 04:23:03.274557 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 25 04:23:03.274557 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 25 04:23:03.274557 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 25 04:23:03.274557 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 25 04:23:03.274557 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 25 04:23:03.274557 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 04:23:03.274557 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:03.277516 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 298/606] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Sep 25 04:23:03 2022] GET /identity/v3/projects/service => generated 88 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:03.278118 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8f1c388-46a4-4d69-82c1-3266ab97e1b4 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:03.324213 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:03.334806 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88adacc0-c773-417a-8a42-c4a45e307f1a None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:03.370123 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f3efd6b-9c0e-437e-8a90-352113b0f991 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:03.370762 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f3efd6b-9c0e-437e-8a90-352113b0f991 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:03.371040 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f3efd6b-9c0e-437e-8a90-352113b0f991 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:03.371308 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f3efd6b-9c0e-437e-8a90-352113b0f991 None admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:03.372167 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f3efd6b-9c0e-437e-8a90-352113b0f991 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:03.372278 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f3efd6b-9c0e-437e-8a90-352113b0f991 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:03.373763 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f3efd6b-9c0e-437e-8a90-352113b0f991 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:03.391951 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 309/607] 10.222.0.25 () {56 vars in 1148 bytes} [Sun Sep 25 04:23:03 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:03.391951 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f3efd6b-9c0e-437e-8a90-352113b0f991 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:03.407915 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8f1c388-46a4-4d69-82c1-3266ab97e1b4 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:03.447343 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7ee4d6f-ea96-40b4-8c15-8d0db52ce05a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:03.448350 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7ee4d6f-ea96-40b4-8c15-8d0db52ce05a None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:03.448977 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7ee4d6f-ea96-40b4-8c15-8d0db52ce05a None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:03.449534 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7ee4d6f-ea96-40b4-8c15-8d0db52ce05a None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:03.450920 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7ee4d6f-ea96-40b4-8c15-8d0db52ce05a None admin] RBAC: Adding query filter params (name=service) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:03.451477 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7ee4d6f-ea96-40b4-8c15-8d0db52ce05a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:03.453463 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7ee4d6f-ea96-40b4-8c15-8d0db52ce05a None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:03.464050 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 299/608] 10.222.0.25 () {56 vars in 1103 bytes} [Sun Sep 25 04:23:03 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:03.468296 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7ee4d6f-ea96-40b4-8c15-8d0db52ce05a None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:03.473258 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:03.500695 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f3efd6b-9c0e-437e-8a90-352113b0f991 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:03.534970 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-592a1885-11a6-4ff9-838f-b06fcd4e3ceb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:03.535574 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-592a1885-11a6-4ff9-838f-b06fcd4e3ceb None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:03.535854 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-592a1885-11a6-4ff9-838f-b06fcd4e3ceb None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:03.536123 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-592a1885-11a6-4ff9-838f-b06fcd4e3ceb None admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:03.536925 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-592a1885-11a6-4ff9-838f-b06fcd4e3ceb None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:03.537300 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-592a1885-11a6-4ff9-838f-b06fcd4e3ceb None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:03.538690 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-592a1885-11a6-4ff9-838f-b06fcd4e3ceb None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:03.551856 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 310/609] 10.222.0.25 () {56 vars in 1148 bytes} [Sun Sep 25 04:23:03 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:03.552340 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-592a1885-11a6-4ff9-838f-b06fcd4e3ceb None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:03.564286 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:03.592994 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7ee4d6f-ea96-40b4-8c15-8d0db52ce05a None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:03.631824 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81ead531-b07a-49e9-b9e5-6761a531d976 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:03.632999 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81ead531-b07a-49e9-b9e5-6761a531d976 None admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:03.633490 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81ead531-b07a-49e9-b9e5-6761a531d976 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:03.633985 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81ead531-b07a-49e9-b9e5-6761a531d976 None admin] PATH_INFO: `/v3/projects/056ca1b1a9974a7581550de0d9217823/users/e703b42c53974daaa608b0ccc3f018b1/roles/9b44520f482c4423bd10568d0f86718c` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:03.651584 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81ead531-b07a-49e9-b9e5-6761a531d976 None admin] RBAC: Authorizing `identity:create_grant(project_id=056ca1b1a9974a7581550de0d9217823, user_id=e703b42c53974daaa608b0ccc3f018b1, role_id=9b44520f482c4423bd10568d0f86718c)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:03.653583 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81ead531-b07a-49e9-b9e5-6761a531d976 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:03.655042 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-592a1885-11a6-4ff9-838f-b06fcd4e3ceb None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:03.671861 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 300/610] 10.222.0.25 () {58 vars in 1439 bytes} [Sun Sep 25 04:23:03 2022] PUT /identity/v3/projects/056ca1b1a9974a7581550de0d9217823/users/e703b42c53974daaa608b0ccc3f018b1/roles/9b44520f482c4423bd10568d0f86718c => generated 0 bytes in 207 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:23:03.673224 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81ead531-b07a-49e9-b9e5-6761a531d976 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:03.718676 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2824b7cb-13da-408f-aa14-6503f6b7f60a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:03.719117 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2824b7cb-13da-408f-aa14-6503f6b7f60a None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:03.719117 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2824b7cb-13da-408f-aa14-6503f6b7f60a None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:03.719277 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2824b7cb-13da-408f-aa14-6503f6b7f60a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:03.743415 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2824b7cb-13da-408f-aa14-6503f6b7f60a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:03.744609 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:03.745010 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2824b7cb-13da-408f-aa14-6503f6b7f60a None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:03.772625 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-2824b7cb-13da-408f-aa14-6503f6b7f60a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:23:03.772625 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:03.772625 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:03.772625 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:03.772625 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:03.772625 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:03.772625 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:03.772625 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:03.772625 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:03.772625 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:03.772625 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:03.772625 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:03.772625 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 04:23:03.772625 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 04:23:03.772625 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 04:23:03.772625 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 04:23:03.773695 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:03.773695 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:03.773695 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:03.773695 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:03.773695 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:03.773695 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:03.773695 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:03.773695 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:03.773695 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:03.773695 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:03.773695 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:03.773695 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:03.773695 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:03.773695 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:03.773695 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:03.773695 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:03.773695 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 04:23:03.775185 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 04:23:03.775185 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:23:03.775185 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:23:03.775185 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 311/611] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:23:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 222 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:03.781981 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2824b7cb-13da-408f-aa14-6503f6b7f60a None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:03.800586 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81ead531-b07a-49e9-b9e5-6761a531d976 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:03.842632 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02956dd5-a456-4f27-bdcc-938c89ef52f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:03.843752 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02956dd5-a456-4f27-bdcc-938c89ef52f7 None admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:03.844373 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02956dd5-a456-4f27-bdcc-938c89ef52f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:03.844963 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02956dd5-a456-4f27-bdcc-938c89ef52f7 None admin] PATH_INFO: `/v3/projects/056ca1b1a9974a7581550de0d9217823/users/ffbc136c77b84e8c87128f7247f4ec03/roles/9b44520f482c4423bd10568d0f86718c` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:03.864886 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02956dd5-a456-4f27-bdcc-938c89ef52f7 None admin] RBAC: Authorizing `identity:create_grant(project_id=056ca1b1a9974a7581550de0d9217823, user_id=ffbc136c77b84e8c87128f7247f4ec03, role_id=9b44520f482c4423bd10568d0f86718c)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:03.866693 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02956dd5-a456-4f27-bdcc-938c89ef52f7 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:03.889633 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 301/612] 10.222.0.25 () {58 vars in 1439 bytes} [Sun Sep 25 04:23:03 2022] PUT /identity/v3/projects/056ca1b1a9974a7581550de0d9217823/users/ffbc136c77b84e8c87128f7247f4ec03/roles/9b44520f482c4423bd10568d0f86718c => generated 0 bytes in 217 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:23:03.912086 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:03.979358 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2824b7cb-13da-408f-aa14-6503f6b7f60a None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:04.020864 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-966da1ff-2b74-488f-9008-0aa60a5bd45a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:04.021848 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-966da1ff-2b74-488f-9008-0aa60a5bd45a None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:04.022338 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-966da1ff-2b74-488f-9008-0aa60a5bd45a None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:04.022814 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-966da1ff-2b74-488f-9008-0aa60a5bd45a None admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:04.023740 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-966da1ff-2b74-488f-9008-0aa60a5bd45a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:04.024265 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-966da1ff-2b74-488f-9008-0aa60a5bd45a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:04.025894 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-966da1ff-2b74-488f-9008-0aa60a5bd45a None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:04.041204 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 312/613] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Sep 25 04:23:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 259 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:04.047570 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02956dd5-a456-4f27-bdcc-938c89ef52f7 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:04.168683 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02956dd5-a456-4f27-bdcc-938c89ef52f7 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:04.172012 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:04.210520 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90a631c5-2a77-40c5-89dd-1b5b48cabaf5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:04.211308 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a631c5-2a77-40c5-89dd-1b5b48cabaf5 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:04.211686 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a631c5-2a77-40c5-89dd-1b5b48cabaf5 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:04.212053 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a631c5-2a77-40c5-89dd-1b5b48cabaf5 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:04.221335 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90a631c5-2a77-40c5-89dd-1b5b48cabaf5 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:04.222931 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90a631c5-2a77-40c5-89dd-1b5b48cabaf5 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:04.235614 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-90a631c5-2a77-40c5-89dd-1b5b48cabaf5 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 25 04:23:04.235614 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:04.235614 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:04.235614 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:04.235614 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:04.235614 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:04.235614 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:04.235614 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:04.235614 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:04.235614 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:04.235614 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:04.235614 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:04.235614 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 25 04:23:04.235614 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 25 04:23:04.235614 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 25 04:23:04.235614 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 25 04:23:04.236864 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:04.236864 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:04.236864 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 25 04:23:04.236864 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 04:23:04.236864 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 25 04:23:04.236864 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 04:23:04.236864 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:04.236864 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:04.236864 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:04.236864 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:04.236864 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:04.236864 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:04.236864 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:04.236864 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:04.236864 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:04.236864 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:04.236864 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:04.236864 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:04.238079 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:04.238079 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:04.238079 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 25 04:23:04.238079 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 25 04:23:04.238079 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 25 04:23:04.238079 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 25 04:23:04.238079 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 25 04:23:04.238079 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 25 04:23:04.238079 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 25 04:23:04.238079 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:04.239670 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 302/614] 10.222.0.25 () {56 vars in 1096 bytes} [Sun Sep 25 04:23:04 2022] GET /identity/v3/users/swift => generated 83 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 04:23:04.245454 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-966da1ff-2b74-488f-9008-0aa60a5bd45a None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:04.377869 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-966da1ff-2b74-488f-9008-0aa60a5bd45a None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:04.409280 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:04.425334 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d725191-2462-486a-99d4-0c83ec55344a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:04.425962 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d725191-2462-486a-99d4-0c83ec55344a None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:04.426248 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d725191-2462-486a-99d4-0c83ec55344a None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:04.426527 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d725191-2462-486a-99d4-0c83ec55344a None admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:04.427274 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d725191-2462-486a-99d4-0c83ec55344a None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:04.427587 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d725191-2462-486a-99d4-0c83ec55344a None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:04.429048 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d725191-2462-486a-99d4-0c83ec55344a None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:04.447611 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 313/615] 10.222.0.25 () {56 vars in 1135 bytes} [Sun Sep 25 04:23:04 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:04.453636 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90a631c5-2a77-40c5-89dd-1b5b48cabaf5 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:04.557923 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90a631c5-2a77-40c5-89dd-1b5b48cabaf5 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:04.593954 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fddb802-f0aa-45fc-8b01-12ba48f90ccd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:04.594619 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fddb802-f0aa-45fc-8b01-12ba48f90ccd None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:04.594904 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fddb802-f0aa-45fc-8b01-12ba48f90ccd None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:04.595179 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fddb802-f0aa-45fc-8b01-12ba48f90ccd None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:04.606131 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fddb802-f0aa-45fc-8b01-12ba48f90ccd None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:04.607662 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fddb802-f0aa-45fc-8b01-12ba48f90ccd None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:04.623192 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-1fddb802-f0aa-45fc-8b01-12ba48f90ccd None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:23:04.623192 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:04.623192 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:04.623192 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:04.623192 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:04.623192 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:04.623192 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:04.623192 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:04.623192 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:04.623192 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:04.623192 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:04.623192 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:04.623192 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 04:23:04.623192 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 04:23:04.623192 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 04:23:04.623192 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 04:23:04.624198 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:04.624198 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:04.624198 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:04.624198 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:04.624198 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:04.624198 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:04.624198 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:04.624198 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:04.624198 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:04.624198 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:04.624198 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:04.624198 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:04.624198 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:04.624198 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:04.624198 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:04.624198 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:04.624198 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 04:23:04.627052 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 04:23:04.627052 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:23:04.627052 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:04.627052 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 303/616] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:23:04 2022] GET /identity/v3/domains/Default => generated 87 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:04.630865 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d725191-2462-486a-99d4-0c83ec55344a None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:04.732293 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d725191-2462-486a-99d4-0c83ec55344a None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:04.782074 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96e04de0-0b53-4008-809f-135f330b375c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:04.782706 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96e04de0-0b53-4008-809f-135f330b375c None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:04.782988 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96e04de0-0b53-4008-809f-135f330b375c None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:04.783264 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96e04de0-0b53-4008-809f-135f330b375c None admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:04.783920 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96e04de0-0b53-4008-809f-135f330b375c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:04.784225 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96e04de0-0b53-4008-809f-135f330b375c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:04.785631 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96e04de0-0b53-4008-809f-135f330b375c None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:04.804091 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 314/617] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Sep 25 04:23:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:04.809726 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fddb802-f0aa-45fc-8b01-12ba48f90ccd None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:04.832933 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:04.922738 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fddb802-f0aa-45fc-8b01-12ba48f90ccd None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:04.947018 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcaa6974-cee5-4bfa-b1a1-18c8af40cdc5 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:04.947332 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcaa6974-cee5-4bfa-b1a1-18c8af40cdc5 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:04.947565 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcaa6974-cee5-4bfa-b1a1-18c8af40cdc5 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:04.949206 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 315/618] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:23:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:23:04.954669 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbe95f5d-b16c-40f1-a723-b62e7d31099e None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:04.954949 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbe95f5d-b16c-40f1-a723-b62e7d31099e None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:04.955226 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbe95f5d-b16c-40f1-a723-b62e7d31099e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:04.959877 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79909099-13b6-484e-9619-02c68c989e19 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:04.960509 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79909099-13b6-484e-9619-02c68c989e19 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:04.960828 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79909099-13b6-484e-9619-02c68c989e19 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:04.961104 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79909099-13b6-484e-9619-02c68c989e19 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:04.969586 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79909099-13b6-484e-9619-02c68c989e19 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:04.971063 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79909099-13b6-484e-9619-02c68c989e19 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:04.974127 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-cbe95f5d-b16c-40f1-a723-b62e7d31099e None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:23:04.975325 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-cbe95f5d-b16c-40f1-a723-b62e7d31099e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:23:04.983271 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-79909099-13b6-484e-9619-02c68c989e19 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 04:23:04.983271 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:04.983271 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:04.983271 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:04.983271 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:04.983271 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:04.983271 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:04.983271 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:04.983271 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:04.983271 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:04.983271 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:04.983271 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:04.983271 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 25 04:23:04.983271 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 25 04:23:04.983271 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 25 04:23:04.985698 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 25 04:23:04.985698 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:04.985698 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:04.985698 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:04.985698 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:04.985698 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:04.985698 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:04.985698 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:04.985698 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:04.985698 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:04.985698 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:04.985698 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:04.985698 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:04.985698 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:04.985698 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:04.985698 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:04.985698 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:04.987844 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 25 04:23:04.987844 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 25 04:23:04.987844 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 25 04:23:04.987844 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 25 04:23:04.987844 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 25 04:23:04.987844 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 25 04:23:04.987844 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 04:23:04.987844 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:04.987844 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 304/619] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Sep 25 04:23:04 2022] GET /identity/v3/projects/service => generated 88 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 04:23:04.993897 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79909099-13b6-484e-9619-02c68c989e19 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:05.036825 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:05.083609 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:05.106835 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 316/620] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:23:04 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 25 04:23:05.109864 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79909099-13b6-484e-9619-02c68c989e19 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:05.146269 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5befa205-5b89-466c-9087-686e4e29d775 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:05.146910 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5befa205-5b89-466c-9087-686e4e29d775 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:05.147193 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5befa205-5b89-466c-9087-686e4e29d775 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:05.147470 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5befa205-5b89-466c-9087-686e4e29d775 None admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:05.148323 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5befa205-5b89-466c-9087-686e4e29d775 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:05.148658 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5befa205-5b89-466c-9087-686e4e29d775 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:05.150180 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5befa205-5b89-466c-9087-686e4e29d775 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:05.170095 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 305/621] 10.222.0.25 () {56 vars in 1148 bytes} [Sun Sep 25 04:23:04 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:05.181713 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbe95f5d-b16c-40f1-a723-b62e7d31099e None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:05.309966 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbe95f5d-b16c-40f1-a723-b62e7d31099e None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:05.344542 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd626ca0-ed88-46e0-a228-8e1fa6b9ab2d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:05.345159 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd626ca0-ed88-46e0-a228-8e1fa6b9ab2d None admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:05.345438 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd626ca0-ed88-46e0-a228-8e1fa6b9ab2d None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:05.345713 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd626ca0-ed88-46e0-a228-8e1fa6b9ab2d None admin] PATH_INFO: `/v3/projects/056ca1b1a9974a7581550de0d9217823/users/23ea88f46e7740d295148a93ade29dd9/roles/9b44520f482c4423bd10568d0f86718c` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:05.360726 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd626ca0-ed88-46e0-a228-8e1fa6b9ab2d None admin] RBAC: Authorizing `identity:create_grant(project_id=056ca1b1a9974a7581550de0d9217823, user_id=23ea88f46e7740d295148a93ade29dd9, role_id=9b44520f482c4423bd10568d0f86718c)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:05.362150 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd626ca0-ed88-46e0-a228-8e1fa6b9ab2d None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:05.380119 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 317/622] 10.222.0.25 () {58 vars in 1439 bytes} [Sun Sep 25 04:23:05 2022] PUT /identity/v3/projects/056ca1b1a9974a7581550de0d9217823/users/23ea88f46e7740d295148a93ade29dd9/roles/9b44520f482c4423bd10568d0f86718c => generated 0 bytes in 199 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:23:05.422684 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5befa205-5b89-466c-9087-686e4e29d775 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:05.535062 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5befa205-5b89-466c-9087-686e4e29d775 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:05.582554 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f4c9cbd-af54-4214-be3d-ec48145df767 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:05.583289 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f4c9cbd-af54-4214-be3d-ec48145df767 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:05.583576 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f4c9cbd-af54-4214-be3d-ec48145df767 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:05.583851 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f4c9cbd-af54-4214-be3d-ec48145df767 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:05.586996 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:05.598269 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f4c9cbd-af54-4214-be3d-ec48145df767 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 04:23:05.605813 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-0f4c9cbd-af54-4214-be3d-ec48145df767 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 25 04:23:05.605813 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:05.605813 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:05.605813 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:05.605813 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:05.605813 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:05.605813 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:05.605813 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:05.605813 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:05.605813 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:05.605813 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:05.605813 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:05.605813 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 04:23:05.605813 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 04:23:05.605813 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 04:23:05.605813 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 04:23:05.608051 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 04:23:05.608051 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 04:23:05.608051 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 04:23:05.608051 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 04:23:05.608051 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:05.608051 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:05.608051 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:05.608051 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:05.608051 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:05.608051 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:05.608051 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:05.608051 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:05.608051 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:05.608051 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:05.608051 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:05.608051 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:05.610261 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:05.610261 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:05.610261 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:05.610261 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:05.610261 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 04:23:05.610261 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 04:23:05.610261 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 04:23:05.610261 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 04:23:05.610261 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 04:23:05.610261 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 04:23:05.610261 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 25 04:23:05.610261 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:05.610261 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 306/623] 10.222.0.25 () {56 vars in 1102 bytes} [Sun Sep 25 04:23:05 2022] GET /identity/v3/roles/service => generated 85 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:05.613203 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd626ca0-ed88-46e0-a228-8e1fa6b9ab2d None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:05.650572 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:05.747149 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd626ca0-ed88-46e0-a228-8e1fa6b9ab2d None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:05.789775 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2ea927c-0a7f-40bb-97ea-d53d5684741f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:05.790455 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2ea927c-0a7f-40bb-97ea-d53d5684741f None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:05.790740 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2ea927c-0a7f-40bb-97ea-d53d5684741f None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:05.791014 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2ea927c-0a7f-40bb-97ea-d53d5684741f None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:05.791737 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2ea927c-0a7f-40bb-97ea-d53d5684741f None admin] RBAC: Adding query filter params (name=service) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:05.792050 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2ea927c-0a7f-40bb-97ea-d53d5684741f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:05.793453 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2ea927c-0a7f-40bb-97ea-d53d5684741f None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:05.802790 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 318/624] 10.222.0.25 () {56 vars in 1103 bytes} [Sun Sep 25 04:23:05 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:05.808673 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f4c9cbd-af54-4214-be3d-ec48145df767 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:05.882348 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d95c13eb-207c-4ae7-8b46-f29866a79dd6 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:05.882667 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d95c13eb-207c-4ae7-8b46-f29866a79dd6 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:05.882825 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d95c13eb-207c-4ae7-8b46-f29866a79dd6 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:05.884143 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 319/625] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:23:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 25 04:23:05.889407 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ba8c63-ffbd-4cec-b3f8-17336ced02d9 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:05.889689 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ba8c63-ffbd-4cec-b3f8-17336ced02d9 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:05.889964 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ba8c63-ffbd-4cec-b3f8-17336ced02d9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:05.909578 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-85ba8c63-ffbd-4cec-b3f8-17336ced02d9 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:23:05.910959 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-85ba8c63-ffbd-4cec-b3f8-17336ced02d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:23:05.925192 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f4c9cbd-af54-4214-be3d-ec48145df767 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:05.988974 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 320/626] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:23:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 25 04:23:05.997481 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58166674-2a90-4c1f-94e8-ecc3925583eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:05.998355 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58166674-2a90-4c1f-94e8-ecc3925583eb None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:05.998785 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58166674-2a90-4c1f-94e8-ecc3925583eb None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:05.999224 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58166674-2a90-4c1f-94e8-ecc3925583eb None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:06.010668 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58166674-2a90-4c1f-94e8-ecc3925583eb None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:06.012351 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58166674-2a90-4c1f-94e8-ecc3925583eb None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:06.034876 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-58166674-2a90-4c1f-94e8-ecc3925583eb None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:23:06.034876 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:06.034876 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:06.034876 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:06.034876 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:06.034876 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:06.034876 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:06.034876 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:06.034876 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:06.034876 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:06.034876 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:06.034876 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:06.034876 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 04:23:06.034876 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 04:23:06.034876 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 04:23:06.034876 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 04:23:06.036135 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:06.036135 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:06.036135 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:06.036135 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:06.036135 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:06.036135 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:06.036135 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:06.036135 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:06.036135 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:06.036135 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:06.036135 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:06.036135 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:06.036135 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:06.036135 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:06.036135 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:06.036135 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:06.036135 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 04:23:06.037337 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 04:23:06.037337 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:23:06.037337 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:06.039124 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 307/627] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:23:05 2022] GET /identity/v3/domains/Default => generated 87 bytes in 231 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:06.044384 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85ba8c63-ffbd-4cec-b3f8-17336ced02d9 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:06.065063 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:06.197428 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85ba8c63-ffbd-4cec-b3f8-17336ced02d9 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:06.246733 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac59d5f7-1398-4dd6-92fe-f304b1677bcf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:06.247354 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac59d5f7-1398-4dd6-92fe-f304b1677bcf None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:06.247630 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac59d5f7-1398-4dd6-92fe-f304b1677bcf None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:06.247898 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac59d5f7-1398-4dd6-92fe-f304b1677bcf None admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:06.248543 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac59d5f7-1398-4dd6-92fe-f304b1677bcf None admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:06.248866 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac59d5f7-1398-4dd6-92fe-f304b1677bcf None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:06.250814 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac59d5f7-1398-4dd6-92fe-f304b1677bcf None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:06.266995 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 321/628] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Sep 25 04:23:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 223 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:06.272629 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58166674-2a90-4c1f-94e8-ecc3925583eb None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:06.324610 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac59d5f7-1398-4dd6-92fe-f304b1677bcf None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:06.351560 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:06.400384 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58166674-2a90-4c1f-94e8-ecc3925583eb None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:06.440776 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d9704ad-e9d3-48f5-8268-a803b9717587 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:06.441560 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d9704ad-e9d3-48f5-8268-a803b9717587 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:06.441764 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d9704ad-e9d3-48f5-8268-a803b9717587 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:06.442091 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d9704ad-e9d3-48f5-8268-a803b9717587 None admin] PATH_INFO: `/v3/users/glance` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:06.450338 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d9704ad-e9d3-48f5-8268-a803b9717587 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:06.452175 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d9704ad-e9d3-48f5-8268-a803b9717587 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:06.470804 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-4d9704ad-e9d3-48f5-8268-a803b9717587 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Sep 25 04:23:06.470804 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:06.470804 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:06.470804 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:06.470804 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:06.470804 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:06.470804 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:06.470804 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:06.470804 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:06.470804 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:06.470804 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:06.470804 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:06.470804 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 25 04:23:06.470804 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 25 04:23:06.470804 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 25 04:23:06.470804 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 25 04:23:06.471937 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:06.471937 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:06.471937 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 25 04:23:06.471937 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 04:23:06.471937 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 25 04:23:06.471937 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 04:23:06.471937 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:06.471937 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:06.471937 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:06.471937 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:06.471937 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:06.471937 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:06.471937 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:06.471937 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:06.471937 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:06.471937 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:06.471937 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:06.471937 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:06.472962 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:06.472962 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:06.472962 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 25 04:23:06.472962 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 25 04:23:06.472962 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 25 04:23:06.472962 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 25 04:23:06.472962 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 25 04:23:06.472962 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 25 04:23:06.472962 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Sep 25 04:23:06.472962 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:06.482792 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 308/629] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Sep 25 04:23:06 2022] GET /identity/v3/users/glance => generated 84 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:06.483759 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d9704ad-e9d3-48f5-8268-a803b9717587 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:06.537067 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:06.585979 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac59d5f7-1398-4dd6-92fe-f304b1677bcf None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:06.637703 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e212584f-3e85-4d72-a6f0-3142e62fd2b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:06.638370 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e212584f-3e85-4d72-a6f0-3142e62fd2b9 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:06.638652 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e212584f-3e85-4d72-a6f0-3142e62fd2b9 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:06.638997 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e212584f-3e85-4d72-a6f0-3142e62fd2b9 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:06.664988 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e212584f-3e85-4d72-a6f0-3142e62fd2b9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 04:23:06.672394 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-e212584f-3e85-4d72-a6f0-3142e62fd2b9 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 25 04:23:06.672394 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:06.672394 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:06.672394 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:06.672394 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:06.672394 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:06.672394 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:06.672394 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:06.672394 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:06.672394 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:06.672394 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:06.672394 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:06.672394 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 04:23:06.672394 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 04:23:06.672394 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 04:23:06.672394 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 04:23:06.673460 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 04:23:06.673460 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 04:23:06.673460 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 04:23:06.673460 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 04:23:06.673460 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:06.673460 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:06.673460 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:06.673460 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:06.673460 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:06.673460 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:06.673460 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:06.673460 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:06.673460 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:06.673460 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:06.673460 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:06.673460 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:06.674326 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:06.674326 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:06.674326 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:06.674326 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:06.674326 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 04:23:06.674326 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 04:23:06.674326 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 04:23:06.674326 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 04:23:06.674326 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 04:23:06.674326 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 04:23:06.674326 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 25 04:23:06.674326 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:23:06.675444 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 322/630] 10.222.0.25 () {56 vars in 1102 bytes} [Sun Sep 25 04:23:06 2022] GET /identity/v3/roles/service => generated 85 bytes in 350 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 04:23:06.684582 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e212584f-3e85-4d72-a6f0-3142e62fd2b9 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:06.693638 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d9704ad-e9d3-48f5-8268-a803b9717587 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:06.711725 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:06.739349 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c4c8828-23e5-4014-bf44-8473bfd23689 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:06.740194 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c4c8828-23e5-4014-bf44-8473bfd23689 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:06.740507 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c4c8828-23e5-4014-bf44-8473bfd23689 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:06.740888 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c4c8828-23e5-4014-bf44-8473bfd23689 None admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:06.741614 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c4c8828-23e5-4014-bf44-8473bfd23689 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:06.741926 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c4c8828-23e5-4014-bf44-8473bfd23689 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:06.743520 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c4c8828-23e5-4014-bf44-8473bfd23689 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:06.764497 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 309/631] 10.222.0.25 () {56 vars in 1137 bytes} [Sun Sep 25 04:23:06 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 281 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:06.765861 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-618ff32f-815f-4465-9141-87e6b7b90e83 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:06.766230 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-618ff32f-815f-4465-9141-87e6b7b90e83 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:06.766506 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-618ff32f-815f-4465-9141-87e6b7b90e83 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:06.767835 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 310/632] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:23:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:23:06.768745 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6cd2966-5d4e-4433-a31a-34dfca443157 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:06.769029 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6cd2966-5d4e-4433-a31a-34dfca443157 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:06.769304 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6cd2966-5d4e-4433-a31a-34dfca443157 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:06.774520 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 311/633] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:23:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:23:06.774520 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6cd2966-5d4e-4433-a31a-34dfca443157 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:06.810758 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e212584f-3e85-4d72-a6f0-3142e62fd2b9 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:06.842835 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:06.848249 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abc42c8b-b92d-4e22-babb-35fd4c41615a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:06.849174 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc42c8b-b92d-4e22-babb-35fd4c41615a None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:06.849811 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc42c8b-b92d-4e22-babb-35fd4c41615a None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:06.850085 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc42c8b-b92d-4e22-babb-35fd4c41615a None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:06.850750 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abc42c8b-b92d-4e22-babb-35fd4c41615a None admin] RBAC: Adding query filter params (name=service) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:06.851053 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abc42c8b-b92d-4e22-babb-35fd4c41615a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:06.852469 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abc42c8b-b92d-4e22-babb-35fd4c41615a None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:06.861175 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 323/634] 10.222.0.25 () {56 vars in 1103 bytes} [Sun Sep 25 04:23:06 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:06.862326 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e296fdde-77fd-4b7a-8eb1-c62426312afb None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:06.862612 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e296fdde-77fd-4b7a-8eb1-c62426312afb None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:06.862886 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e296fdde-77fd-4b7a-8eb1-c62426312afb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:06.882922 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-e296fdde-77fd-4b7a-8eb1-c62426312afb None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:23:06.883413 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6cd2966-5d4e-4433-a31a-34dfca443157 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:06.884672 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-e296fdde-77fd-4b7a-8eb1-c62426312afb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:23:06.920655 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91587080-3454-431f-b152-f940d9b076b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:06.922012 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91587080-3454-431f-b152-f940d9b076b9 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:06.922344 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91587080-3454-431f-b152-f940d9b076b9 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:06.923391 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91587080-3454-431f-b152-f940d9b076b9 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:06.956323 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 324/635] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:23:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 04:23:06.957884 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f49d55-a2d2-4adc-94d9-4813235f1939 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:06.958394 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f49d55-a2d2-4adc-94d9-4813235f1939 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:06.958870 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f49d55-a2d2-4adc-94d9-4813235f1939 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:06.974497 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:06.978513 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-c9f49d55-a2d2-4adc-94d9-4813235f1939 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:23:06.979942 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-c9f49d55-a2d2-4adc-94d9-4813235f1939 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:23:07.011789 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91587080-3454-431f-b152-f940d9b076b9 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:07.013610 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91587080-3454-431f-b152-f940d9b076b9 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:07.068785 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 325/636] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:23:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 04:23:07.070249 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9f49d55-a2d2-4adc-94d9-4813235f1939 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:07.147879 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:07.188975 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 312/637] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Sep 25 04:23:06 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 417 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 25 04:23:07.195995 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91587080-3454-431f-b152-f940d9b076b9 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:07.199500 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9f49d55-a2d2-4adc-94d9-4813235f1939 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:07.239635 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47ced441-2355-4028-bc53-8f1b3ecdbbc7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:07.240281 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ced441-2355-4028-bc53-8f1b3ecdbbc7 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:07.240570 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ced441-2355-4028-bc53-8f1b3ecdbbc7 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:07.241259 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ced441-2355-4028-bc53-8f1b3ecdbbc7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:07.253225 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47ced441-2355-4028-bc53-8f1b3ecdbbc7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:07.254681 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47ced441-2355-4028-bc53-8f1b3ecdbbc7 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:07.272103 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-47ced441-2355-4028-bc53-8f1b3ecdbbc7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:23:07.272103 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:07.272103 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:07.272103 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:07.272103 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:07.272103 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:07.272103 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:07.272103 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:07.272103 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:07.272103 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:07.272103 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:07.272103 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:07.272103 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 04:23:07.272103 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 04:23:07.272103 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 04:23:07.272103 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 04:23:07.273121 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:07.273121 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:07.273121 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:07.273121 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:07.273121 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:07.273121 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:07.273121 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:07.273121 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:07.273121 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:07.273121 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:07.273121 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:07.273121 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:07.273121 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:07.273121 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:07.273121 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:07.273121 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:07.273121 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 04:23:07.274057 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 04:23:07.274057 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:23:07.274057 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:23:07.274057 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 326/638] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:23:07 2022] GET /identity/v3/domains/Default => generated 87 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:07.279062 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47ced441-2355-4028-bc53-8f1b3ecdbbc7 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:07.317223 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:07.357061 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91587080-3454-431f-b152-f940d9b076b9 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:07.392239 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47ced441-2355-4028-bc53-8f1b3ecdbbc7 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:07.400005 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:07.405599 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72229688-540a-4fb8-8aa0-df0c81d58f1d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:07.406961 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72229688-540a-4fb8-8aa0-df0c81d58f1d None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:07.407455 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72229688-540a-4fb8-8aa0-df0c81d58f1d None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:07.408138 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72229688-540a-4fb8-8aa0-df0c81d58f1d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:07.419492 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72229688-540a-4fb8-8aa0-df0c81d58f1d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:07.421182 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72229688-540a-4fb8-8aa0-df0c81d58f1d None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:07.433264 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac8a9320-a1c7-47c4-93df-a448e3798fa4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:07.433876 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac8a9320-a1c7-47c4-93df-a448e3798fa4 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:07.434157 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac8a9320-a1c7-47c4-93df-a448e3798fa4 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:07.434426 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac8a9320-a1c7-47c4-93df-a448e3798fa4 None admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:07.435067 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac8a9320-a1c7-47c4-93df-a448e3798fa4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:07.435370 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac8a9320-a1c7-47c4-93df-a448e3798fa4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:07.436468 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-72229688-540a-4fb8-8aa0-df0c81d58f1d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:23:07.436468 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:07.436468 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:07.436468 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:07.436468 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:07.436468 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:07.436468 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:07.436468 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:07.436468 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:07.436468 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:07.436468 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:07.436468 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:07.436468 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 04:23:07.436468 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 04:23:07.436468 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 04:23:07.436468 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 04:23:07.437514 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:07.437514 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:07.437514 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:07.437514 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:07.437514 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:07.437514 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:07.437514 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:07.437514 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:07.437514 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:07.437514 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:07.437514 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:07.437514 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:07.437514 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:07.437514 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:07.437514 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:07.437514 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:07.437514 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 04:23:07.443997 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 04:23:07.443997 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:23:07.443997 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:07.444192 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac8a9320-a1c7-47c4-93df-a448e3798fa4 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:07.444275 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 313/639] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:23:07 2022] GET /identity/v3/domains/Default => generated 87 bytes in 242 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:07.444275 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72229688-540a-4fb8-8aa0-df0c81d58f1d None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:07.460030 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 327/640] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Sep 25 04:23:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:07.460030 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac8a9320-a1c7-47c4-93df-a448e3798fa4 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:07.551317 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72229688-540a-4fb8-8aa0-df0c81d58f1d None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:07.570025 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:07.570192 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac8a9320-a1c7-47c4-93df-a448e3798fa4 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:07.591744 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd1b086f-447b-419e-a369-c55b58be0e03 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:07.592382 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd1b086f-447b-419e-a369-c55b58be0e03 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:07.592691 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd1b086f-447b-419e-a369-c55b58be0e03 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:07.593081 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd1b086f-447b-419e-a369-c55b58be0e03 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:07.604188 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd1b086f-447b-419e-a369-c55b58be0e03 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 04:23:07.606478 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b51d0df-d1b5-4c6b-9fda-95542cf0ee77 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:07.607122 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b51d0df-d1b5-4c6b-9fda-95542cf0ee77 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:07.607399 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b51d0df-d1b5-4c6b-9fda-95542cf0ee77 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:07.607668 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b51d0df-d1b5-4c6b-9fda-95542cf0ee77 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:07.610136 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-cd1b086f-447b-419e-a369-c55b58be0e03 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 25 04:23:07.610136 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:07.610136 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:07.610136 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:07.610136 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:07.610136 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:07.610136 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:07.610136 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:07.610136 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:07.610136 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:07.610136 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:07.610136 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:07.610136 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 04:23:07.610136 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 04:23:07.610136 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 04:23:07.610136 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 04:23:07.611071 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 04:23:07.611071 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 04:23:07.611071 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 04:23:07.611071 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 04:23:07.611071 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:07.611071 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:07.611071 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:07.611071 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:07.611071 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:07.611071 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:07.611071 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:07.611071 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:07.611071 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:07.611071 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:07.611071 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:07.611071 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:07.611944 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:07.611944 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:07.611944 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:07.611944 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:07.611944 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 04:23:07.611944 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 04:23:07.611944 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 04:23:07.611944 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 04:23:07.611944 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 04:23:07.611944 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 04:23:07.611944 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 25 04:23:07.611944 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:07.611944 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 314/641] 10.222.0.25 () {56 vars in 1102 bytes} [Sun Sep 25 04:23:07 2022] GET /identity/v3/roles/service => generated 85 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:07.611944 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd1b086f-447b-419e-a369-c55b58be0e03 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:07.625164 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b51d0df-d1b5-4c6b-9fda-95542cf0ee77 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 04:23:07.630107 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-7b51d0df-d1b5-4c6b-9fda-95542cf0ee77 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 25 04:23:07.630107 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:07.630107 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:07.630107 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:07.630107 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:07.630107 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:07.630107 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:07.630107 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:07.630107 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:07.630107 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:07.630107 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:07.630107 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:07.630107 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 04:23:07.630107 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 04:23:07.630107 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 04:23:07.630107 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 04:23:07.631156 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 04:23:07.631156 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 04:23:07.631156 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 04:23:07.631156 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 04:23:07.631156 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:07.631156 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:07.631156 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:07.631156 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:07.631156 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:07.631156 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:07.631156 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:07.631156 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:07.631156 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:07.631156 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:07.631156 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:07.631156 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:07.632553 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:07.632553 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:07.632553 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:07.632553 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:07.632553 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 04:23:07.632553 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 04:23:07.632553 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 04:23:07.632553 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 04:23:07.632553 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 04:23:07.632553 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 04:23:07.632553 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 25 04:23:07.632553 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:23:07.632553 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 328/642] 10.222.0.25 () {56 vars in 1102 bytes} [Sun Sep 25 04:23:07 2022] GET /identity/v3/roles/service => generated 85 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:07.632553 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b51d0df-d1b5-4c6b-9fda-95542cf0ee77 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:07.661417 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:07.727191 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd1b086f-447b-419e-a369-c55b58be0e03 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:07.736658 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b51d0df-d1b5-4c6b-9fda-95542cf0ee77 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:07.762891 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d66309d-01e3-432b-a541-f637144cbf0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:07.763507 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d66309d-01e3-432b-a541-f637144cbf0e None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:07.763789 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d66309d-01e3-432b-a541-f637144cbf0e None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:07.764240 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d66309d-01e3-432b-a541-f637144cbf0e None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:07.764729 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d66309d-01e3-432b-a541-f637144cbf0e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:07.765034 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d66309d-01e3-432b-a541-f637144cbf0e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:07.766761 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d66309d-01e3-432b-a541-f637144cbf0e None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:07.772562 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-807a6a2c-409c-4fcc-b168-b92a7e85b1a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:07.773223 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-807a6a2c-409c-4fcc-b168-b92a7e85b1a7 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:07.773504 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-807a6a2c-409c-4fcc-b168-b92a7e85b1a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:07.773776 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-807a6a2c-409c-4fcc-b168-b92a7e85b1a7 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:07.779614 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 315/643] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Sep 25 04:23:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:07.780705 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d66309d-01e3-432b-a541-f637144cbf0e None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:07.785947 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-807a6a2c-409c-4fcc-b168-b92a7e85b1a7 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:07.787635 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-807a6a2c-409c-4fcc-b168-b92a7e85b1a7 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:07.800063 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-807a6a2c-409c-4fcc-b168-b92a7e85b1a7 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Sep 25 04:23:07.800063 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:07.800063 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:07.800063 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:07.800063 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:07.800063 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:07.800063 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:07.800063 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:07.800063 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:07.800063 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:07.800063 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:07.800063 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:07.800063 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 25 04:23:07.800063 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 25 04:23:07.800063 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 25 04:23:07.800063 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 25 04:23:07.801129 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:07.801129 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:07.801129 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 25 04:23:07.801129 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 04:23:07.801129 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 25 04:23:07.801129 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 04:23:07.801129 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:07.801129 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:07.801129 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:07.801129 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:07.801129 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:07.801129 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:07.801129 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:07.801129 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:07.801129 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:07.801129 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:07.801129 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:07.801129 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:07.802313 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:07.802313 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:07.802313 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 25 04:23:07.802313 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 25 04:23:07.802313 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 25 04:23:07.802313 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 25 04:23:07.802313 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 25 04:23:07.802313 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 25 04:23:07.802313 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Sep 25 04:23:07.802313 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:23:07.805994 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 329/644] 10.222.0.25 () {56 vars in 1102 bytes} [Sun Sep 25 04:23:07 2022] GET /identity/v3/users/neutron => generated 85 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:07.806863 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-807a6a2c-409c-4fcc-b168-b92a7e85b1a7 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:07.828941 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:07.896988 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d66309d-01e3-432b-a541-f637144cbf0e None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:07.904197 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:07.921492 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-807a6a2c-409c-4fcc-b168-b92a7e85b1a7 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:07.932663 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94a5d2e8-8e52-47e6-bfb0-679ddf06b2d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:07.933281 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94a5d2e8-8e52-47e6-bfb0-679ddf06b2d0 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:07.933563 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94a5d2e8-8e52-47e6-bfb0-679ddf06b2d0 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:07.933835 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94a5d2e8-8e52-47e6-bfb0-679ddf06b2d0 None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:07.934611 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94a5d2e8-8e52-47e6-bfb0-679ddf06b2d0 None admin] RBAC: Adding query filter params (name=service) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:07.934918 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94a5d2e8-8e52-47e6-bfb0-679ddf06b2d0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:07.936423 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94a5d2e8-8e52-47e6-bfb0-679ddf06b2d0 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:07.945857 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 316/645] 10.222.0.25 () {56 vars in 1103 bytes} [Sun Sep 25 04:23:07 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:07.946656 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94a5d2e8-8e52-47e6-bfb0-679ddf06b2d0 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:07.966674 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b7b0616-a8f6-4bbb-a39f-6f943b039577 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:07.967292 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b7b0616-a8f6-4bbb-a39f-6f943b039577 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:07.967571 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b7b0616-a8f6-4bbb-a39f-6f943b039577 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:07.967847 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b7b0616-a8f6-4bbb-a39f-6f943b039577 None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:07.969182 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b7b0616-a8f6-4bbb-a39f-6f943b039577 None admin] RBAC: Adding query filter params (name=service) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:07.969491 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b7b0616-a8f6-4bbb-a39f-6f943b039577 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:07.970865 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b7b0616-a8f6-4bbb-a39f-6f943b039577 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:07.979192 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 330/646] 10.222.0.25 () {56 vars in 1103 bytes} [Sun Sep 25 04:23:07 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:07.979983 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b7b0616-a8f6-4bbb-a39f-6f943b039577 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:08.048167 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:08.051104 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94a5d2e8-8e52-47e6-bfb0-679ddf06b2d0 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:08.084243 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b7b0616-a8f6-4bbb-a39f-6f943b039577 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:08.086652 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15427cc5-3b45-4274-8f16-2c3b5900c821 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:08.087289 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15427cc5-3b45-4274-8f16-2c3b5900c821 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:08.087571 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15427cc5-3b45-4274-8f16-2c3b5900c821 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:08.087845 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15427cc5-3b45-4274-8f16-2c3b5900c821 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:08.095259 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15427cc5-3b45-4274-8f16-2c3b5900c821 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:08.096778 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15427cc5-3b45-4274-8f16-2c3b5900c821 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:08.108831 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-15427cc5-3b45-4274-8f16-2c3b5900c821 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 04:23:08.108831 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:08.108831 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:08.108831 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:08.108831 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:08.108831 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:08.108831 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:08.108831 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:08.108831 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:08.108831 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:08.108831 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:08.108831 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:08.108831 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 25 04:23:08.108831 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 25 04:23:08.108831 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 25 04:23:08.110375 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 25 04:23:08.110375 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:08.110375 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:08.110375 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:08.110375 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:08.110375 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:08.110375 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:08.110375 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:08.110375 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:08.110375 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:08.110375 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:08.110375 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:08.110375 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:08.110375 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:08.110375 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:08.110375 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:08.110375 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:08.111336 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 25 04:23:08.111336 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 25 04:23:08.111336 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 25 04:23:08.111336 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 25 04:23:08.111336 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 25 04:23:08.111336 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 25 04:23:08.111336 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 04:23:08.111336 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:08.111336 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 317/647] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Sep 25 04:23:07 2022] GET /identity/v3/projects/service => generated 88 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:08.111336 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15427cc5-3b45-4274-8f16-2c3b5900c821 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:08.121718 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b52e091-5543-4446-bd1d-97a8f7255cf5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:08.122327 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b52e091-5543-4446-bd1d-97a8f7255cf5 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:08.122610 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b52e091-5543-4446-bd1d-97a8f7255cf5 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:08.122889 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b52e091-5543-4446-bd1d-97a8f7255cf5 None admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:08.123620 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b52e091-5543-4446-bd1d-97a8f7255cf5 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:08.123926 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b52e091-5543-4446-bd1d-97a8f7255cf5 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:08.125412 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b52e091-5543-4446-bd1d-97a8f7255cf5 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:08.142820 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 331/648] 10.222.0.25 () {56 vars in 1139 bytes} [Sun Sep 25 04:23:07 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:08.143959 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d71a70a0-61ae-4ab4-98ec-5c9119e0508f None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:08.144252 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d71a70a0-61ae-4ab4-98ec-5c9119e0508f None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:08.144526 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d71a70a0-61ae-4ab4-98ec-5c9119e0508f None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:08.145768 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 332/649] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:23:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:23:08.146458 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d71a70a0-61ae-4ab4-98ec-5c9119e0508f None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:08.155382 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:08.214785 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15427cc5-3b45-4274-8f16-2c3b5900c821 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:08.247582 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d71a70a0-61ae-4ab4-98ec-5c9119e0508f None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:08.247749 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a3c3dca-9c69-4050-a0b9-3f79ef9524eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:08.248233 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a3c3dca-9c69-4050-a0b9-3f79ef9524eb None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:08.248521 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a3c3dca-9c69-4050-a0b9-3f79ef9524eb None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:08.248826 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a3c3dca-9c69-4050-a0b9-3f79ef9524eb None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:08.259096 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a3c3dca-9c69-4050-a0b9-3f79ef9524eb None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:08.260558 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a3c3dca-9c69-4050-a0b9-3f79ef9524eb None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:08.275298 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-7a3c3dca-9c69-4050-a0b9-3f79ef9524eb None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:23:08.275298 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:08.275298 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:08.275298 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:08.275298 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:08.275298 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:08.275298 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:08.275298 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:08.275298 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:08.275298 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:08.275298 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:08.275298 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:08.275298 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 04:23:08.275298 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 04:23:08.275298 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 04:23:08.275298 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 04:23:08.276369 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:08.276369 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:08.276369 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:08.276369 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:08.276369 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:08.276369 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:08.276369 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:08.276369 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:08.276369 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:08.276369 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:08.276369 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:08.276369 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:08.276369 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:08.276369 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:08.276369 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:08.276369 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:08.276369 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 04:23:08.277342 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 04:23:08.277342 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:23:08.277342 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:08.277342 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 318/650] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:23:08 2022] GET /identity/v3/domains/Default => generated 87 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:08.277342 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a3c3dca-9c69-4050-a0b9-3f79ef9524eb None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:08.286116 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c367cb4-1488-4b8c-a1fa-f3db6b3b2cc8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:08.286749 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c367cb4-1488-4b8c-a1fa-f3db6b3b2cc8 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:08.287027 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c367cb4-1488-4b8c-a1fa-f3db6b3b2cc8 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:08.287298 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c367cb4-1488-4b8c-a1fa-f3db6b3b2cc8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:08.291528 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:08.298500 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c367cb4-1488-4b8c-a1fa-f3db6b3b2cc8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:08.299950 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c367cb4-1488-4b8c-a1fa-f3db6b3b2cc8 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:08.314642 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-8c367cb4-1488-4b8c-a1fa-f3db6b3b2cc8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:23:08.314642 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:08.314642 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:08.314642 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:08.314642 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:08.314642 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:08.314642 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:08.314642 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:08.314642 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:08.314642 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:08.314642 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:08.314642 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:08.314642 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 04:23:08.314642 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 04:23:08.314642 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 04:23:08.314642 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 04:23:08.315660 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:08.315660 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:08.315660 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:08.315660 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:08.315660 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:08.315660 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:08.315660 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:08.315660 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:08.315660 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:08.315660 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:08.315660 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:08.315660 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:08.315660 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:08.315660 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:08.315660 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:08.315660 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:08.315660 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 04:23:08.316580 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 04:23:08.316580 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:23:08.316580 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:23:08.316580 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 333/651] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:23:08 2022] GET /identity/v3/domains/Default => generated 87 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:08.316580 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c367cb4-1488-4b8c-a1fa-f3db6b3b2cc8 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:08.376776 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:08.388360 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a3c3dca-9c69-4050-a0b9-3f79ef9524eb None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:08.421977 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52bc4089-006c-4986-ad7d-38830c9ea063 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:08.422584 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52bc4089-006c-4986-ad7d-38830c9ea063 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:08.422871 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52bc4089-006c-4986-ad7d-38830c9ea063 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:08.423146 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52bc4089-006c-4986-ad7d-38830c9ea063 None admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:08.423827 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52bc4089-006c-4986-ad7d-38830c9ea063 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:08.423927 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c367cb4-1488-4b8c-a1fa-f3db6b3b2cc8 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:08.424139 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52bc4089-006c-4986-ad7d-38830c9ea063 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:08.425606 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52bc4089-006c-4986-ad7d-38830c9ea063 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:08.437938 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 319/652] 10.222.0.25 () {56 vars in 1148 bytes} [Sun Sep 25 04:23:08 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:08.439100 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b953a08c-df93-47de-9337-f991af583771 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:08.439383 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b953a08c-df93-47de-9337-f991af583771 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:08.439653 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b953a08c-df93-47de-9337-f991af583771 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:08.457684 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-b953a08c-df93-47de-9337-f991af583771 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:23:08.458287 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f6bda83-6e86-4985-b0d4-ad84b0069189 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:08.458722 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f6bda83-6e86-4985-b0d4-ad84b0069189 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:08.458921 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-b953a08c-df93-47de-9337-f991af583771 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:23:08.459016 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f6bda83-6e86-4985-b0d4-ad84b0069189 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:08.459254 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f6bda83-6e86-4985-b0d4-ad84b0069189 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:08.526540 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f6bda83-6e86-4985-b0d4-ad84b0069189 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:08.527983 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f6bda83-6e86-4985-b0d4-ad84b0069189 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:08.530023 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 320/653] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:23:08 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 04:23:08.530929 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b953a08c-df93-47de-9337-f991af583771 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:08.533293 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:08.604548 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:08.632630 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b953a08c-df93-47de-9337-f991af583771 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:08.656027 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 334/654] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Sep 25 04:23:08 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 340 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 25 04:23:08.657514 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f6bda83-6e86-4985-b0d4-ad84b0069189 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:08.672055 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad9c8b9c-c260-4cc7-b264-570be3254d9d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:08.672780 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad9c8b9c-c260-4cc7-b264-570be3254d9d None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:08.673078 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad9c8b9c-c260-4cc7-b264-570be3254d9d None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:08.673334 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad9c8b9c-c260-4cc7-b264-570be3254d9d None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:08.673978 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad9c8b9c-c260-4cc7-b264-570be3254d9d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:08.674284 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad9c8b9c-c260-4cc7-b264-570be3254d9d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:08.675656 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad9c8b9c-c260-4cc7-b264-570be3254d9d None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:08.688718 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 321/655] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Sep 25 04:23:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:08.689519 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad9c8b9c-c260-4cc7-b264-570be3254d9d None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:08.755740 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:08.769509 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f6bda83-6e86-4985-b0d4-ad84b0069189 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:08.798069 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad9c8b9c-c260-4cc7-b264-570be3254d9d None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:08.814999 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61b68147-8d2d-4dc0-95e7-f2288d9373d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:08.815613 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61b68147-8d2d-4dc0-95e7-f2288d9373d7 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:08.815891 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61b68147-8d2d-4dc0-95e7-f2288d9373d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:08.816159 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61b68147-8d2d-4dc0-95e7-f2288d9373d7 None admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:08.816838 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61b68147-8d2d-4dc0-95e7-f2288d9373d7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:08.817142 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61b68147-8d2d-4dc0-95e7-f2288d9373d7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:08.818504 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61b68147-8d2d-4dc0-95e7-f2288d9373d7 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:08.830562 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 335/656] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Sep 25 04:23:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:08.831355 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61b68147-8d2d-4dc0-95e7-f2288d9373d7 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:08.832502 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17184068-3a56-4422-8a89-c9892279ee48 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:08.833144 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17184068-3a56-4422-8a89-c9892279ee48 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:08.833480 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17184068-3a56-4422-8a89-c9892279ee48 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:08.833770 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17184068-3a56-4422-8a89-c9892279ee48 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:08.834504 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17184068-3a56-4422-8a89-c9892279ee48 None admin] RBAC: Adding query filter params (role.id=9b44520f482c4423bd10568d0f86718c, scope.project.id=056ca1b1a9974a7581550de0d9217823, user.id=316eaa67d5464091af47efd489b96b87) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:08.834936 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17184068-3a56-4422-8a89-c9892279ee48 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:08.836606 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17184068-3a56-4422-8a89-c9892279ee48 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:08.842472 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 322/657] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Sep 25 04:23:08 2022] GET /identity/v3/role_assignments?user.id=316eaa67d5464091af47efd489b96b87&scope.project.id=056ca1b1a9974a7581550de0d9217823&role.id=9b44520f482c4423bd10568d0f86718c => generated 608 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:08.843181 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17184068-3a56-4422-8a89-c9892279ee48 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:08.854022 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:08.946637 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61b68147-8d2d-4dc0-95e7-f2288d9373d7 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:08.958680 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17184068-3a56-4422-8a89-c9892279ee48 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:08.986943 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3e2c68d-6c5a-46ef-9606-1e2f60f2747e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:08.987312 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:08.987776 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e2c68d-6c5a-46ef-9606-1e2f60f2747e None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:08.988305 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e2c68d-6c5a-46ef-9606-1e2f60f2747e None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:08.988417 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e2c68d-6c5a-46ef-9606-1e2f60f2747e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:08.999199 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c851c45-8675-402b-86fd-0ab0b5b2ee98 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:09.000224 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c851c45-8675-402b-86fd-0ab0b5b2ee98 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:09.000536 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3e2c68d-6c5a-46ef-9606-1e2f60f2747e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:09.000673 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c851c45-8675-402b-86fd-0ab0b5b2ee98 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:09.000888 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c851c45-8675-402b-86fd-0ab0b5b2ee98 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:09.001826 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3e2c68d-6c5a-46ef-9606-1e2f60f2747e None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:09.009347 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c851c45-8675-402b-86fd-0ab0b5b2ee98 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:09.010814 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c851c45-8675-402b-86fd-0ab0b5b2ee98 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:09.018990 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-b3e2c68d-6c5a-46ef-9606-1e2f60f2747e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:23:09.018990 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:09.018990 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:09.018990 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:09.018990 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:09.018990 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:09.018990 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:09.018990 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:09.018990 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:09.018990 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:09.018990 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:09.018990 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:09.018990 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 04:23:09.018990 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 04:23:09.018990 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 04:23:09.018990 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 04:23:09.020176 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:09.020176 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:09.020176 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:09.020176 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:09.020176 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:09.020176 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:09.020176 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:09.020176 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:09.020176 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:09.020176 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:09.020176 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:09.020176 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:09.020176 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:09.020176 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:09.020176 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:09.020176 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:09.020176 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 04:23:09.021176 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 04:23:09.021176 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:23:09.021176 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:23:09.021176 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 336/658] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:23:08 2022] GET /identity/v3/domains/Default => generated 87 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:09.021176 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3e2c68d-6c5a-46ef-9606-1e2f60f2747e None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:09.030991 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-3c851c45-8675-402b-86fd-0ab0b5b2ee98 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 25 04:23:09.030991 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:09.030991 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:09.030991 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:09.030991 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:09.030991 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:09.030991 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:09.030991 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:09.030991 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:09.030991 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:09.030991 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:09.030991 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:09.030991 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 25 04:23:09.030991 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 25 04:23:09.030991 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 25 04:23:09.030991 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 25 04:23:09.032100 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:09.032100 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:09.032100 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 25 04:23:09.032100 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 04:23:09.032100 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 25 04:23:09.032100 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 04:23:09.032100 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:09.032100 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:09.032100 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:09.032100 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:09.032100 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:09.032100 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:09.032100 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:09.032100 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:09.032100 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:09.032100 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:09.032100 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:09.032100 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:09.034298 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:09.034298 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:09.034298 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 25 04:23:09.034298 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 25 04:23:09.034298 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 25 04:23:09.034298 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 25 04:23:09.034298 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 25 04:23:09.034298 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 25 04:23:09.034298 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 25 04:23:09.034298 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:09.034298 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 323/659] 10.222.0.25 () {56 vars in 1093 bytes} [Sun Sep 25 04:23:08 2022] GET /identity/v3/users/nova => generated 82 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:09.034298 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c851c45-8675-402b-86fd-0ab0b5b2ee98 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:09.107153 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:09.143494 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3e2c68d-6c5a-46ef-9606-1e2f60f2747e None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:09.146405 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c851c45-8675-402b-86fd-0ab0b5b2ee98 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:09.179435 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63d2de44-421a-4cbe-b6d8-05eae9f216dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:09.180185 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63d2de44-421a-4cbe-b6d8-05eae9f216dd None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:09.180473 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63d2de44-421a-4cbe-b6d8-05eae9f216dd None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:09.180780 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63d2de44-421a-4cbe-b6d8-05eae9f216dd None admin] PATH_INFO: `/v3/roles/service` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:09.181328 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75640f90-3786-4fcb-8e13-d027349b625a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:09.181960 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75640f90-3786-4fcb-8e13-d027349b625a None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:09.182240 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75640f90-3786-4fcb-8e13-d027349b625a None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:09.182509 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75640f90-3786-4fcb-8e13-d027349b625a None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:09.191012 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75640f90-3786-4fcb-8e13-d027349b625a None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:09.191851 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63d2de44-421a-4cbe-b6d8-05eae9f216dd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 04:23:09.192462 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75640f90-3786-4fcb-8e13-d027349b625a None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:09.197262 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-63d2de44-421a-4cbe-b6d8-05eae9f216dd None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 25 04:23:09.197262 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:09.197262 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:09.197262 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:09.197262 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:09.197262 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:09.197262 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:09.197262 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:09.197262 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:09.197262 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:09.197262 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:09.197262 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:09.197262 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 04:23:09.197262 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 04:23:09.197262 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 04:23:09.197262 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 04:23:09.198216 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 04:23:09.198216 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 04:23:09.198216 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 04:23:09.198216 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 04:23:09.198216 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:09.198216 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:09.198216 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:09.198216 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:09.198216 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:09.198216 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:09.198216 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:09.198216 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:09.198216 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:09.198216 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:09.198216 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:09.198216 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:09.201700 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:09.201700 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:09.201700 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:09.201700 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:09.201700 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 04:23:09.201700 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 04:23:09.201700 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 04:23:09.201700 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 04:23:09.201700 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 04:23:09.201700 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 04:23:09.201700 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 25 04:23:09.201700 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:23:09.201700 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 337/660] 10.222.0.25 () {56 vars in 1102 bytes} [Sun Sep 25 04:23:09 2022] GET /identity/v3/roles/service => generated 85 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:09.201700 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63d2de44-421a-4cbe-b6d8-05eae9f216dd None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:09.209446 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-75640f90-3786-4fcb-8e13-d027349b625a None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Sep 25 04:23:09.209446 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:09.209446 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:09.209446 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:09.209446 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:09.209446 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:09.209446 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:09.209446 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:09.209446 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:09.209446 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:09.209446 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:09.209446 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:09.209446 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 25 04:23:09.209446 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 25 04:23:09.209446 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 25 04:23:09.209446 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 25 04:23:09.210419 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:09.210419 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:09.210419 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 25 04:23:09.210419 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 04:23:09.210419 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 25 04:23:09.210419 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 04:23:09.210419 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:09.210419 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:09.210419 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:09.210419 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:09.210419 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:09.210419 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:09.210419 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:09.210419 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:09.210419 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:09.210419 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:09.210419 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:09.210419 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:09.211385 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:09.211385 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:09.211385 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 25 04:23:09.211385 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 25 04:23:09.211385 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 25 04:23:09.211385 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 25 04:23:09.211385 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 25 04:23:09.211385 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 25 04:23:09.211385 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Sep 25 04:23:09.211385 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:09.211385 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 324/661] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Sep 25 04:23:09 2022] GET /identity/v3/users/cinder => generated 84 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:09.212554 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75640f90-3786-4fcb-8e13-d027349b625a None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:09.260868 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:09.317906 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75640f90-3786-4fcb-8e13-d027349b625a None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:09.318592 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63d2de44-421a-4cbe-b6d8-05eae9f216dd None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:09.351339 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:09.356194 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1947e95e-73ca-45d6-9162-270ae9ae721a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:09.356983 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1947e95e-73ca-45d6-9162-270ae9ae721a None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:09.357426 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1947e95e-73ca-45d6-9162-270ae9ae721a None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:09.357773 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1947e95e-73ca-45d6-9162-270ae9ae721a None admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:09.358062 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c54545b2-f2ae-4b98-bdb4-7c2378da04d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:09.358659 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1947e95e-73ca-45d6-9162-270ae9ae721a None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:09.358763 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c54545b2-f2ae-4b98-bdb4-7c2378da04d6 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:09.359140 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1947e95e-73ca-45d6-9162-270ae9ae721a None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:09.359407 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c54545b2-f2ae-4b98-bdb4-7c2378da04d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:09.359771 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c54545b2-f2ae-4b98-bdb4-7c2378da04d6 None admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:09.360485 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c54545b2-f2ae-4b98-bdb4-7c2378da04d6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:09.360889 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c54545b2-f2ae-4b98-bdb4-7c2378da04d6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:09.361093 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1947e95e-73ca-45d6-9162-270ae9ae721a None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:09.362369 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c54545b2-f2ae-4b98-bdb4-7c2378da04d6 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:09.380056 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 338/662] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Sep 25 04:23:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:09.380056 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c54545b2-f2ae-4b98-bdb4-7c2378da04d6 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:09.384599 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 325/663] 10.222.0.25 () {56 vars in 1133 bytes} [Sun Sep 25 04:23:09 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:09.388709 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1947e95e-73ca-45d6-9162-270ae9ae721a None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:09.496038 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1947e95e-73ca-45d6-9162-270ae9ae721a None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:09.499272 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:09.503236 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c54545b2-f2ae-4b98-bdb4-7c2378da04d6 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:09.533718 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-748cde99-9991-4f34-b844-e9e7fc941d7f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:09.534319 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-748cde99-9991-4f34-b844-e9e7fc941d7f None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:09.534601 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-748cde99-9991-4f34-b844-e9e7fc941d7f None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:09.534871 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-748cde99-9991-4f34-b844-e9e7fc941d7f None admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:09.535588 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-748cde99-9991-4f34-b844-e9e7fc941d7f None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:09.535893 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-748cde99-9991-4f34-b844-e9e7fc941d7f None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:09.537352 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-748cde99-9991-4f34-b844-e9e7fc941d7f None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:09.538614 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-577b0fca-fa8a-4d9c-8c93-59092ac1702e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:09.539309 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-577b0fca-fa8a-4d9c-8c93-59092ac1702e None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:09.539679 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-577b0fca-fa8a-4d9c-8c93-59092ac1702e None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:09.540052 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-577b0fca-fa8a-4d9c-8c93-59092ac1702e None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:09.540834 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-577b0fca-fa8a-4d9c-8c93-59092ac1702e None admin] RBAC: Adding query filter params (name=service) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:09.541232 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-577b0fca-fa8a-4d9c-8c93-59092ac1702e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:09.542665 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-577b0fca-fa8a-4d9c-8c93-59092ac1702e None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:09.552002 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 339/664] 10.222.0.25 () {56 vars in 1103 bytes} [Sun Sep 25 04:23:09 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:09.552972 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-577b0fca-fa8a-4d9c-8c93-59092ac1702e None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:09.565719 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 326/665] 10.222.0.25 () {56 vars in 1137 bytes} [Sun Sep 25 04:23:09 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:09.565719 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-748cde99-9991-4f34-b844-e9e7fc941d7f None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:09.614268 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:09.666653 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-748cde99-9991-4f34-b844-e9e7fc941d7f None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:09.669994 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-577b0fca-fa8a-4d9c-8c93-59092ac1702e None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:09.701992 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53a0df83-be1e-440d-9f71-bbf391d1d9d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:09.703307 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a0df83-be1e-440d-9f71-bbf391d1d9d7 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:09.703743 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a0df83-be1e-440d-9f71-bbf391d1d9d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:09.704092 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a0df83-be1e-440d-9f71-bbf391d1d9d7 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:09.708892 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c541c88e-eef9-459a-8eef-3ec02a689e8a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:09.709602 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c541c88e-eef9-459a-8eef-3ec02a689e8a None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:09.709952 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c541c88e-eef9-459a-8eef-3ec02a689e8a None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:09.710292 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c541c88e-eef9-459a-8eef-3ec02a689e8a None admin] PATH_INFO: `/v3/projects/service` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:09.717591 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c541c88e-eef9-459a-8eef-3ec02a689e8a None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:09.719538 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c541c88e-eef9-459a-8eef-3ec02a689e8a None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:09.731173 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-c541c88e-eef9-459a-8eef-3ec02a689e8a None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 04:23:09.731173 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:09.731173 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:09.731173 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:09.731173 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:09.731173 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:09.731173 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:09.731173 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:09.731173 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:09.731173 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:09.731173 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:09.731173 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:09.731173 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 25 04:23:09.731173 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 25 04:23:09.731173 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 25 04:23:09.732269 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 25 04:23:09.732269 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:09.732269 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:09.732269 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:09.732269 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:09.732269 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:09.732269 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:09.732269 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:09.732269 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:09.732269 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:09.732269 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:09.732269 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:09.732269 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:09.732269 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:09.732269 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:09.732269 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:09.732269 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:09.733401 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 25 04:23:09.733401 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 25 04:23:09.733401 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 25 04:23:09.733401 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 25 04:23:09.733401 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 25 04:23:09.733401 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 25 04:23:09.733401 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 04:23:09.733401 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:23:09.733401 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 340/666] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Sep 25 04:23:09 2022] GET /identity/v3/projects/service => generated 88 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:09.734782 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c541c88e-eef9-459a-8eef-3ec02a689e8a None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:09.741898 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:09.782687 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53a0df83-be1e-440d-9f71-bbf391d1d9d7 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:09.784249 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53a0df83-be1e-440d-9f71-bbf391d1d9d7 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:09.844469 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c541c88e-eef9-459a-8eef-3ec02a689e8a None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:09.857556 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:09.883335 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7e23967-4689-4262-9ed7-6ef275897e5e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:09.884119 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7e23967-4689-4262-9ed7-6ef275897e5e None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:09.884402 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7e23967-4689-4262-9ed7-6ef275897e5e None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:09.884701 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7e23967-4689-4262-9ed7-6ef275897e5e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:09.897115 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 327/667] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Sep 25 04:23:09 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 331 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 25 04:23:09.897115 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53a0df83-be1e-440d-9f71-bbf391d1d9d7 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:09.899814 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7e23967-4689-4262-9ed7-6ef275897e5e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:09.901813 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7e23967-4689-4262-9ed7-6ef275897e5e None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:09.917728 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-a7e23967-4689-4262-9ed7-6ef275897e5e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:23:09.917728 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:09.917728 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:09.917728 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:09.917728 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:09.917728 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:09.917728 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:09.917728 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:09.917728 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:09.917728 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:09.917728 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:09.917728 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:09.917728 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 04:23:09.917728 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 04:23:09.917728 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 04:23:09.917728 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 04:23:09.918808 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:09.918808 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:09.918808 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:09.918808 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:09.918808 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:09.918808 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:09.918808 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:09.918808 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:09.918808 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:09.918808 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:09.918808 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:09.918808 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:09.918808 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:09.918808 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:09.918808 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:09.918808 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:09.918808 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 04:23:09.919738 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 04:23:09.919738 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:23:09.919738 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:23:09.919738 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 341/668] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:23:09 2022] GET /identity/v3/domains/Default => generated 87 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:09.919738 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7e23967-4689-4262-9ed7-6ef275897e5e None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:09.961855 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:10.002681 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53a0df83-be1e-440d-9f71-bbf391d1d9d7 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:10.032618 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7e23967-4689-4262-9ed7-6ef275897e5e None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:10.038134 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-045871e2-7f3b-42b9-b993-cc8a657d3837 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:10.038747 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-045871e2-7f3b-42b9-b993-cc8a657d3837 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:10.039023 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-045871e2-7f3b-42b9-b993-cc8a657d3837 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:10.039291 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-045871e2-7f3b-42b9-b993-cc8a657d3837 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:10.067901 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34dcdbfc-8243-4698-a665-69a762cc6b40 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:10.068502 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34dcdbfc-8243-4698-a665-69a762cc6b40 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:10.068810 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34dcdbfc-8243-4698-a665-69a762cc6b40 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:10.069081 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34dcdbfc-8243-4698-a665-69a762cc6b40 None admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:10.069767 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34dcdbfc-8243-4698-a665-69a762cc6b40 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:10.070072 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34dcdbfc-8243-4698-a665-69a762cc6b40 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:10.071580 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34dcdbfc-8243-4698-a665-69a762cc6b40 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:10.085906 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 342/669] 10.222.0.25 () {56 vars in 1148 bytes} [Sun Sep 25 04:23:09 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:10.091413 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34dcdbfc-8243-4698-a665-69a762cc6b40 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:10.112424 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-045871e2-7f3b-42b9-b993-cc8a657d3837 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:10.114072 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-045871e2-7f3b-42b9-b993-cc8a657d3837 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:10.121098 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:10.178370 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:10.204592 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34dcdbfc-8243-4698-a665-69a762cc6b40 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:10.222620 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 328/670] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Sep 25 04:23:09 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 329 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 25 04:23:10.224128 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-045871e2-7f3b-42b9-b993-cc8a657d3837 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:10.247042 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6490f4a4-a70d-427c-ae00-d00122b8b955 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:10.247765 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6490f4a4-a70d-427c-ae00-d00122b8b955 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:10.248120 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6490f4a4-a70d-427c-ae00-d00122b8b955 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:10.248455 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6490f4a4-a70d-427c-ae00-d00122b8b955 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:10.259608 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6490f4a4-a70d-427c-ae00-d00122b8b955 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:10.261109 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6490f4a4-a70d-427c-ae00-d00122b8b955 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:10.276455 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-6490f4a4-a70d-427c-ae00-d00122b8b955 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:23:10.276455 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:10.276455 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:10.276455 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:10.276455 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:10.276455 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:10.276455 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:10.276455 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:10.276455 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:10.276455 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:10.276455 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:10.276455 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:10.276455 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 04:23:10.276455 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 04:23:10.276455 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 04:23:10.276455 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 04:23:10.277637 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:10.277637 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:10.277637 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:10.277637 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:10.277637 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:10.277637 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:10.277637 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:10.277637 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:10.277637 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:10.277637 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:10.277637 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:10.277637 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:10.277637 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:10.277637 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:10.277637 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:10.277637 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:10.277637 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 04:23:10.284137 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 04:23:10.284137 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:23:10.284137 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:23:10.284137 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 343/671] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:23:10 2022] GET /identity/v3/domains/Default => generated 87 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:10.284137 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6490f4a4-a70d-427c-ae00-d00122b8b955 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:10.341285 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-045871e2-7f3b-42b9-b993-cc8a657d3837 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:10.366167 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:10.379996 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8584c8f8-919e-4982-8962-eb3fe21180d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:10.380600 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8584c8f8-919e-4982-8962-eb3fe21180d0 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:10.381017 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8584c8f8-919e-4982-8962-eb3fe21180d0 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:10.381295 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8584c8f8-919e-4982-8962-eb3fe21180d0 None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:10.381958 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8584c8f8-919e-4982-8962-eb3fe21180d0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:10.382264 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8584c8f8-919e-4982-8962-eb3fe21180d0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:10.383642 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8584c8f8-919e-4982-8962-eb3fe21180d0 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:10.391336 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6490f4a4-a70d-427c-ae00-d00122b8b955 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:10.399200 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 329/672] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Sep 25 04:23:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:10.399200 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8584c8f8-919e-4982-8962-eb3fe21180d0 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:10.424478 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:10.428765 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d85031e8-5dc9-4553-ba72-0b9cf241c255 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:10.429626 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d85031e8-5dc9-4553-ba72-0b9cf241c255 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:10.429626 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d85031e8-5dc9-4553-ba72-0b9cf241c255 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:10.430048 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d85031e8-5dc9-4553-ba72-0b9cf241c255 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:10.430847 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d85031e8-5dc9-4553-ba72-0b9cf241c255 None admin] RBAC: Adding query filter params (role.id=9b44520f482c4423bd10568d0f86718c, scope.project.id=056ca1b1a9974a7581550de0d9217823, user.id=0268e88e0c5645a98bab5664be6e49bc) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:10.431233 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d85031e8-5dc9-4553-ba72-0b9cf241c255 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:10.432747 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d85031e8-5dc9-4553-ba72-0b9cf241c255 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:10.448668 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 344/673] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Sep 25 04:23:10 2022] GET /identity/v3/role_assignments?user.id=0268e88e0c5645a98bab5664be6e49bc&scope.project.id=056ca1b1a9974a7581550de0d9217823&role.id=9b44520f482c4423bd10568d0f86718c => generated 608 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:10.448668 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d85031e8-5dc9-4553-ba72-0b9cf241c255 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:10.525332 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8584c8f8-919e-4982-8962-eb3fe21180d0 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:10.563840 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee369e35-6868-49a4-9081-bc64b3ab8156 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:10.564739 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee369e35-6868-49a4-9081-bc64b3ab8156 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:10.565095 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee369e35-6868-49a4-9081-bc64b3ab8156 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:10.565434 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee369e35-6868-49a4-9081-bc64b3ab8156 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:10.571879 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d85031e8-5dc9-4553-ba72-0b9cf241c255 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:10.582452 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee369e35-6868-49a4-9081-bc64b3ab8156 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:10.584344 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee369e35-6868-49a4-9081-bc64b3ab8156 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:10.600987 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-ee369e35-6868-49a4-9081-bc64b3ab8156 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:23:10.600987 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:10.600987 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:10.600987 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:10.600987 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:10.600987 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:10.600987 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:10.600987 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:10.600987 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:10.600987 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:10.600987 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:10.600987 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:10.600987 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 04:23:10.600987 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 04:23:10.600987 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 04:23:10.600987 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 04:23:10.602242 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:10.602242 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:10.602242 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:10.602242 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:10.602242 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:10.602242 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:10.602242 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:10.602242 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:10.602242 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:10.602242 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:10.602242 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:10.602242 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:10.602242 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:10.602242 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:10.602242 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:10.602242 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:10.602242 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 04:23:10.603362 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 04:23:10.603362 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:23:10.603362 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:10.605300 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 330/674] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:23:10 2022] GET /identity/v3/domains/Default => generated 87 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:10.606391 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee369e35-6868-49a4-9081-bc64b3ab8156 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:10.624296 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45ae58af-8bc0-4316-afcb-2c4ced724928 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:10.625226 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45ae58af-8bc0-4316-afcb-2c4ced724928 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:10.625511 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45ae58af-8bc0-4316-afcb-2c4ced724928 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:10.626004 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45ae58af-8bc0-4316-afcb-2c4ced724928 None admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:10.626848 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45ae58af-8bc0-4316-afcb-2c4ced724928 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:10.627304 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45ae58af-8bc0-4316-afcb-2c4ced724928 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:10.628955 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45ae58af-8bc0-4316-afcb-2c4ced724928 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:10.640769 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:10.649890 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 345/675] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Sep 25 04:23:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:10.649890 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45ae58af-8bc0-4316-afcb-2c4ced724928 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:10.709824 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:10.742806 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee369e35-6868-49a4-9081-bc64b3ab8156 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:10.779918 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45ae58af-8bc0-4316-afcb-2c4ced724928 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:10.789307 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3125e2ca-e0c1-49af-adf4-a69251ba0a9e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:10.790589 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3125e2ca-e0c1-49af-adf4-a69251ba0a9e None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:10.791043 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3125e2ca-e0c1-49af-adf4-a69251ba0a9e None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:10.791841 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3125e2ca-e0c1-49af-adf4-a69251ba0a9e None admin] PATH_INFO: `/v3/users/swift` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:10.802797 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3125e2ca-e0c1-49af-adf4-a69251ba0a9e None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:10.804960 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3125e2ca-e0c1-49af-adf4-a69251ba0a9e None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:10.817138 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a04c7f3-c353-42aa-a825-b4d99551c5a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:10.817854 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a04c7f3-c353-42aa-a825-b4d99551c5a4 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:10.818232 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a04c7f3-c353-42aa-a825-b4d99551c5a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:10.818572 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a04c7f3-c353-42aa-a825-b4d99551c5a4 None admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:10.819295 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a04c7f3-c353-42aa-a825-b4d99551c5a4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:10.819672 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a04c7f3-c353-42aa-a825-b4d99551c5a4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:10.821156 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a04c7f3-c353-42aa-a825-b4d99551c5a4 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:10.822340 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-3125e2ca-e0c1-49af-adf4-a69251ba0a9e None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 25 04:23:10.822340 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:10.822340 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:10.822340 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:10.822340 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:10.822340 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:10.822340 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:10.822340 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:10.822340 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:10.822340 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:10.822340 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:10.822340 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:10.822340 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 25 04:23:10.822340 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 25 04:23:10.822340 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 25 04:23:10.822340 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 25 04:23:10.823457 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:10.823457 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:10.823457 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 25 04:23:10.823457 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 04:23:10.823457 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 25 04:23:10.823457 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 04:23:10.823457 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:10.823457 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:10.823457 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:10.823457 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:10.823457 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:10.823457 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:10.823457 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:10.823457 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:10.823457 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:10.823457 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:10.823457 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:10.823457 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:10.824673 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:10.824673 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:10.824673 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 25 04:23:10.824673 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 25 04:23:10.824673 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 25 04:23:10.824673 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 25 04:23:10.824673 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 25 04:23:10.824673 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 25 04:23:10.824673 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 25 04:23:10.824673 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:10.828178 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 331/676] 10.222.0.25 () {56 vars in 1096 bytes} [Sun Sep 25 04:23:10 2022] GET /identity/v3/users/swift => generated 83 bytes in 222 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:10.833762 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3125e2ca-e0c1-49af-adf4-a69251ba0a9e None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:10.841196 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 346/677] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Sep 25 04:23:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:10.842282 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a04c7f3-c353-42aa-a825-b4d99551c5a4 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:10.916074 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:10.961199 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3125e2ca-e0c1-49af-adf4-a69251ba0a9e None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:10.965544 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a04c7f3-c353-42aa-a825-b4d99551c5a4 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:11.008173 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0277003-dcf6-44d7-9b90-bbb4ab8e532a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:11.008919 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0277003-dcf6-44d7-9b90-bbb4ab8e532a None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:11.009208 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0277003-dcf6-44d7-9b90-bbb4ab8e532a None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:11.009489 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0277003-dcf6-44d7-9b90-bbb4ab8e532a None admin] PATH_INFO: `/v3/projects/service` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:11.011687 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:11.013491 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9094dea5-71cc-48ae-b5f4-c8fd6486c8c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:11.014406 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9094dea5-71cc-48ae-b5f4-c8fd6486c8c2 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:11.014887 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9094dea5-71cc-48ae-b5f4-c8fd6486c8c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:11.015413 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9094dea5-71cc-48ae-b5f4-c8fd6486c8c2 None admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:11.017001 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9094dea5-71cc-48ae-b5f4-c8fd6486c8c2 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:11.017997 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9094dea5-71cc-48ae-b5f4-c8fd6486c8c2 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:11.018796 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0277003-dcf6-44d7-9b90-bbb4ab8e532a None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:11.020212 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9094dea5-71cc-48ae-b5f4-c8fd6486c8c2 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:11.020781 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0277003-dcf6-44d7-9b90-bbb4ab8e532a None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:11.041199 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-d0277003-dcf6-44d7-9b90-bbb4ab8e532a None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 04:23:11.041199 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:11.041199 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:11.041199 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:11.041199 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:11.041199 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:11.041199 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:11.041199 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:11.041199 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:11.041199 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:11.041199 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:11.041199 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:11.041199 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 25 04:23:11.041199 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 25 04:23:11.041199 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 25 04:23:11.042346 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 25 04:23:11.042346 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:11.042346 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:11.042346 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:11.042346 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:11.042346 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:11.042346 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:11.042346 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:11.042346 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:11.042346 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:11.042346 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:11.042346 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:11.042346 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:11.042346 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:11.042346 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:11.042346 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:11.042346 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:11.043257 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 25 04:23:11.043257 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 25 04:23:11.043257 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 25 04:23:11.043257 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 25 04:23:11.043257 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 25 04:23:11.043257 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 25 04:23:11.043257 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 04:23:11.043257 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:11.048910 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 332/679] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Sep 25 04:23:10 2022] GET /identity/v3/projects/service => generated 88 bytes in 219 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:11.049131 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 347/679] 10.222.0.25 () {56 vars in 1135 bytes} [Sun Sep 25 04:23:10 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:11.049252 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0277003-dcf6-44d7-9b90-bbb4ab8e532a None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:11.050548 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9094dea5-71cc-48ae-b5f4-c8fd6486c8c2 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:11.157817 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0277003-dcf6-44d7-9b90-bbb4ab8e532a None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:11.167042 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9094dea5-71cc-48ae-b5f4-c8fd6486c8c2 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:11.185687 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:11.193690 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-125015a4-92e8-4423-a5c8-52cdc292fb64 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:11.194323 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-125015a4-92e8-4423-a5c8-52cdc292fb64 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:11.194606 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-125015a4-92e8-4423-a5c8-52cdc292fb64 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:11.194878 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-125015a4-92e8-4423-a5c8-52cdc292fb64 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:11.201736 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-125015a4-92e8-4423-a5c8-52cdc292fb64 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:11.203704 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-125015a4-92e8-4423-a5c8-52cdc292fb64 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:11.205960 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-febca0eb-de88-4f6d-910c-d70d6eaf7453 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:11.206562 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-febca0eb-de88-4f6d-910c-d70d6eaf7453 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:11.206841 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-febca0eb-de88-4f6d-910c-d70d6eaf7453 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:11.207106 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-febca0eb-de88-4f6d-910c-d70d6eaf7453 None admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:11.207785 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-febca0eb-de88-4f6d-910c-d70d6eaf7453 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:11.208087 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-febca0eb-de88-4f6d-910c-d70d6eaf7453 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:11.209504 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-febca0eb-de88-4f6d-910c-d70d6eaf7453 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:11.215668 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-125015a4-92e8-4423-a5c8-52cdc292fb64 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 04:23:11.215668 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:11.215668 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:11.215668 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:11.215668 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:11.215668 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:11.215668 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:11.215668 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:11.215668 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:11.215668 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:11.215668 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:11.215668 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:11.215668 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 25 04:23:11.215668 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 25 04:23:11.215668 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 25 04:23:11.216733 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 25 04:23:11.216733 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:11.216733 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:11.216733 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:11.216733 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:11.216733 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:11.216733 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:11.216733 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:11.216733 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:11.216733 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:11.216733 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:11.216733 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:11.216733 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:11.216733 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:11.216733 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:11.216733 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:11.216733 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:11.217661 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 25 04:23:11.217661 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 25 04:23:11.217661 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 25 04:23:11.217661 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 25 04:23:11.217661 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 25 04:23:11.217661 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 25 04:23:11.217661 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 04:23:11.217661 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:11.217661 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 333/680] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Sep 25 04:23:11 2022] GET /identity/v3/projects/service => generated 88 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:11.217661 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-125015a4-92e8-4423-a5c8-52cdc292fb64 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:11.227104 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 348/681] 10.222.0.25 () {56 vars in 1148 bytes} [Sun Sep 25 04:23:11 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:11.229194 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76874100-21e7-41d4-bd94-d6ddc8b50e6f None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:11.229713 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76874100-21e7-41d4-bd94-d6ddc8b50e6f None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:11.230527 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76874100-21e7-41d4-bd94-d6ddc8b50e6f None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:11.232311 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 349/682] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:23:11 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:23:11.234450 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76874100-21e7-41d4-bd94-d6ddc8b50e6f None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:11.277364 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:11.326914 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-125015a4-92e8-4423-a5c8-52cdc292fb64 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:11.352681 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76874100-21e7-41d4-bd94-d6ddc8b50e6f None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:11.363041 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39cd316d-8f99-4364-95c4-773edd18120c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:11.363961 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39cd316d-8f99-4364-95c4-773edd18120c None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:11.364359 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39cd316d-8f99-4364-95c4-773edd18120c None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:11.364536 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39cd316d-8f99-4364-95c4-773edd18120c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:11.390934 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd9b5bb4-9ae4-4f5a-a679-ca1b8d25bf48 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:11.391825 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd9b5bb4-9ae4-4f5a-a679-ca1b8d25bf48 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:11.392256 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd9b5bb4-9ae4-4f5a-a679-ca1b8d25bf48 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:11.392731 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd9b5bb4-9ae4-4f5a-a679-ca1b8d25bf48 None admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:11.393596 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd9b5bb4-9ae4-4f5a-a679-ca1b8d25bf48 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:11.393906 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd9b5bb4-9ae4-4f5a-a679-ca1b8d25bf48 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:11.395326 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd9b5bb4-9ae4-4f5a-a679-ca1b8d25bf48 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:11.407874 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 350/683] 10.222.0.25 () {56 vars in 1148 bytes} [Sun Sep 25 04:23:11 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:11.408697 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd9b5bb4-9ae4-4f5a-a679-ca1b8d25bf48 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:11.444794 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39cd316d-8f99-4364-95c4-773edd18120c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:11.446243 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39cd316d-8f99-4364-95c4-773edd18120c None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:11.450964 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:11.510092 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:11.529217 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd9b5bb4-9ae4-4f5a-a679-ca1b8d25bf48 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:11.558423 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 334/684] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Sep 25 04:23:11 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 341 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 25 04:23:11.559458 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce0c15b5-5bde-4441-9b8e-3ea2177f6b01 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:11.559742 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce0c15b5-5bde-4441-9b8e-3ea2177f6b01 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:11.560012 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce0c15b5-5bde-4441-9b8e-3ea2177f6b01 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:11.569788 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2ede2af-3689-4d11-9fd9-437dc3532b44 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:11.570471 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2ede2af-3689-4d11-9fd9-437dc3532b44 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:11.570823 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2ede2af-3689-4d11-9fd9-437dc3532b44 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:11.571164 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2ede2af-3689-4d11-9fd9-437dc3532b44 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:11.571976 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2ede2af-3689-4d11-9fd9-437dc3532b44 None admin] RBAC: Adding query filter params (role.id=9b44520f482c4423bd10568d0f86718c, scope.project.id=056ca1b1a9974a7581550de0d9217823, user.id=e703b42c53974daaa608b0ccc3f018b1) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:11.572292 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2ede2af-3689-4d11-9fd9-437dc3532b44 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:11.573753 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2ede2af-3689-4d11-9fd9-437dc3532b44 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:11.578949 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-ce0c15b5-5bde-4441-9b8e-3ea2177f6b01 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:23:11.580080 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-ce0c15b5-5bde-4441-9b8e-3ea2177f6b01 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:23:11.580667 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 351/685] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Sep 25 04:23:11 2022] GET /identity/v3/role_assignments?user.id=e703b42c53974daaa608b0ccc3f018b1&scope.project.id=056ca1b1a9974a7581550de0d9217823&role.id=9b44520f482c4423bd10568d0f86718c => generated 608 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:11.589118 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2ede2af-3689-4d11-9fd9-437dc3532b44 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:11.672500 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 335/686] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:23:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 04:23:11.674508 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce0c15b5-5bde-4441-9b8e-3ea2177f6b01 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:11.710215 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2ede2af-3689-4d11-9fd9-437dc3532b44 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:11.712000 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:11.768919 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69acccb5-86df-4fc8-b66f-699f77d273f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:11.769282 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69acccb5-86df-4fc8-b66f-699f77d273f3 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:11.769534 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69acccb5-86df-4fc8-b66f-699f77d273f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:11.769802 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69acccb5-86df-4fc8-b66f-699f77d273f3 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:11.770533 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69acccb5-86df-4fc8-b66f-699f77d273f3 None admin] RBAC: Adding query filter params (role.id=9b44520f482c4423bd10568d0f86718c, scope.project.id=056ca1b1a9974a7581550de0d9217823, user.id=ffbc136c77b84e8c87128f7247f4ec03) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:11.770839 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69acccb5-86df-4fc8-b66f-699f77d273f3 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:11.772322 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69acccb5-86df-4fc8-b66f-699f77d273f3 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:11.788350 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 352/687] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Sep 25 04:23:11 2022] GET /identity/v3/role_assignments?user.id=ffbc136c77b84e8c87128f7247f4ec03&scope.project.id=056ca1b1a9974a7581550de0d9217823&role.id=9b44520f482c4423bd10568d0f86718c => generated 608 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:11.792017 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:11.812223 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce0c15b5-5bde-4441-9b8e-3ea2177f6b01 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:11.852985 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ad70b6a-e5ca-44b5-b960-02e8da89ffb4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:11.853974 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ad70b6a-e5ca-44b5-b960-02e8da89ffb4 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:11.854483 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ad70b6a-e5ca-44b5-b960-02e8da89ffb4 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:11.854960 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ad70b6a-e5ca-44b5-b960-02e8da89ffb4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:11.868044 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ad70b6a-e5ca-44b5-b960-02e8da89ffb4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:11.869891 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ad70b6a-e5ca-44b5-b960-02e8da89ffb4 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:11.888871 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-9ad70b6a-e5ca-44b5-b960-02e8da89ffb4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:23:11.888871 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:11.888871 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:11.888871 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:11.888871 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:11.888871 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:11.888871 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:11.888871 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:11.888871 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:11.888871 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:11.888871 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:11.888871 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:11.888871 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 04:23:11.888871 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 04:23:11.888871 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 04:23:11.888871 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 04:23:11.891260 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:11.891260 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:11.891260 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:11.891260 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:11.891260 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:11.891260 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:11.891260 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:11.891260 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:11.891260 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:11.891260 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:11.891260 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:11.891260 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:11.891260 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:11.891260 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:11.891260 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:11.891260 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:11.891260 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 04:23:11.892416 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 04:23:11.892416 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:23:11.892416 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:11.894725 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 336/688] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:23:11 2022] GET /identity/v3/domains/Default => generated 87 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:11.901825 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69acccb5-86df-4fc8-b66f-699f77d273f3 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:11.991781 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ad70b6a-e5ca-44b5-b960-02e8da89ffb4 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:12.031244 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69acccb5-86df-4fc8-b66f-699f77d273f3 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:12.079197 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e48148bb-a194-40e9-8c05-72222cf1cb00 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:12.079596 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e48148bb-a194-40e9-8c05-72222cf1cb00 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:12.079870 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e48148bb-a194-40e9-8c05-72222cf1cb00 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:12.080362 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e48148bb-a194-40e9-8c05-72222cf1cb00 None admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:12.081040 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e48148bb-a194-40e9-8c05-72222cf1cb00 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:12.082198 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e48148bb-a194-40e9-8c05-72222cf1cb00 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:12.082725 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e48148bb-a194-40e9-8c05-72222cf1cb00 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:12.096483 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 353/689] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Sep 25 04:23:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:12.103750 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e48148bb-a194-40e9-8c05-72222cf1cb00 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:12.126545 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ad70b6a-e5ca-44b5-b960-02e8da89ffb4 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:12.139038 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:12.174472 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4321ba3f-6e41-449e-9755-1f6ff84f37d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:12.175806 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4321ba3f-6e41-449e-9755-1f6ff84f37d0 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:12.176536 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4321ba3f-6e41-449e-9755-1f6ff84f37d0 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:12.177068 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4321ba3f-6e41-449e-9755-1f6ff84f37d0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:12.180908 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:12.192560 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4321ba3f-6e41-449e-9755-1f6ff84f37d0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:12.194886 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4321ba3f-6e41-449e-9755-1f6ff84f37d0 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:12.216393 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-4321ba3f-6e41-449e-9755-1f6ff84f37d0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:23:12.216393 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:12.216393 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:12.216393 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:12.216393 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:12.216393 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:12.216393 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:12.216393 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:12.216393 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:12.216393 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:12.216393 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:12.216393 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:12.216393 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 04:23:12.216393 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 04:23:12.216393 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 04:23:12.216393 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 04:23:12.217577 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:12.217577 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:12.217577 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:12.217577 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:12.217577 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:12.217577 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:12.217577 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:12.217577 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:12.217577 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:12.217577 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:12.217577 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:12.217577 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:12.217577 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:12.217577 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:12.217577 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:12.217577 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:12.217577 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 04:23:12.219197 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 04:23:12.219197 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:23:12.219197 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:12.221164 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e48148bb-a194-40e9-8c05-72222cf1cb00 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:12.221404 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 337/690] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:23:11 2022] GET /identity/v3/domains/Default => generated 87 bytes in 230 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 04:23:12.240969 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4321ba3f-6e41-449e-9755-1f6ff84f37d0 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:12.261613 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a06d18c-fb2f-4cd6-baab-355116fe478e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:12.262259 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a06d18c-fb2f-4cd6-baab-355116fe478e None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:12.262538 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a06d18c-fb2f-4cd6-baab-355116fe478e None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:12.262806 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a06d18c-fb2f-4cd6-baab-355116fe478e None admin] PATH_INFO: `/v3/projects/service` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:12.272066 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a06d18c-fb2f-4cd6-baab-355116fe478e None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:12.272066 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a06d18c-fb2f-4cd6-baab-355116fe478e None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:12.284053 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-9a06d18c-fb2f-4cd6-baab-355116fe478e None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 04:23:12.284053 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:12.284053 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:12.284053 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:12.284053 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:12.284053 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:12.284053 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:12.284053 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:12.284053 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:12.284053 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:12.284053 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:12.284053 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:12.284053 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 25 04:23:12.284053 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 25 04:23:12.284053 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 25 04:23:12.285219 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 25 04:23:12.285219 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:12.285219 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:12.285219 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:12.285219 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:12.285219 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:12.285219 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:12.285219 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:12.285219 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:12.285219 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:12.285219 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:12.285219 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:12.285219 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:12.285219 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:12.285219 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:12.285219 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:12.285219 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:12.286445 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 25 04:23:12.286445 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 25 04:23:12.286445 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 25 04:23:12.286445 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 25 04:23:12.286445 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 25 04:23:12.286445 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 25 04:23:12.286445 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 04:23:12.286445 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:23:12.292318 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 354/691] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Sep 25 04:23:12 2022] GET /identity/v3/projects/service => generated 88 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:12.293578 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a06d18c-fb2f-4cd6-baab-355116fe478e None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:12.389281 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:12.399099 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a06d18c-fb2f-4cd6-baab-355116fe478e None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:12.434131 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aac5c196-38e7-4c42-8f7b-eee8ac5ed5ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:12.434928 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aac5c196-38e7-4c42-8f7b-eee8ac5ed5ce None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:12.435304 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aac5c196-38e7-4c42-8f7b-eee8ac5ed5ce None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:12.435677 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aac5c196-38e7-4c42-8f7b-eee8ac5ed5ce None admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:12.436477 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aac5c196-38e7-4c42-8f7b-eee8ac5ed5ce None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:12.437451 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aac5c196-38e7-4c42-8f7b-eee8ac5ed5ce None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:12.437688 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4321ba3f-6e41-449e-9755-1f6ff84f37d0 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:12.439211 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aac5c196-38e7-4c42-8f7b-eee8ac5ed5ce None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:12.452839 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 355/692] 10.222.0.25 () {56 vars in 1148 bytes} [Sun Sep 25 04:23:12 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:23:12.460672 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aac5c196-38e7-4c42-8f7b-eee8ac5ed5ce None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:12.521421 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7869eec-5e53-4352-b7b1-faca39bbbac8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:12.522471 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7869eec-5e53-4352-b7b1-faca39bbbac8 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:12.522995 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7869eec-5e53-4352-b7b1-faca39bbbac8 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:12.523416 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7869eec-5e53-4352-b7b1-faca39bbbac8 None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:12.524201 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7869eec-5e53-4352-b7b1-faca39bbbac8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:12.524693 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7869eec-5e53-4352-b7b1-faca39bbbac8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:12.526525 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7869eec-5e53-4352-b7b1-faca39bbbac8 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:12.549090 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 338/693] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Sep 25 04:23:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 309 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:12.554608 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7869eec-5e53-4352-b7b1-faca39bbbac8 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:12.594006 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aac5c196-38e7-4c42-8f7b-eee8ac5ed5ce None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:12.649438 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:12.662377 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a913ad3b-2ae1-4d29-a8c8-144186abfb8b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:12.662998 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a913ad3b-2ae1-4d29-a8c8-144186abfb8b None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:12.663276 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a913ad3b-2ae1-4d29-a8c8-144186abfb8b None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:12.663545 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a913ad3b-2ae1-4d29-a8c8-144186abfb8b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:12.664535 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a913ad3b-2ae1-4d29-a8c8-144186abfb8b None admin] RBAC: Adding query filter params (role.id=9b44520f482c4423bd10568d0f86718c, scope.project.id=056ca1b1a9974a7581550de0d9217823, user.id=23ea88f46e7740d295148a93ade29dd9) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:12.665812 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a913ad3b-2ae1-4d29-a8c8-144186abfb8b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:12.667168 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a913ad3b-2ae1-4d29-a8c8-144186abfb8b None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:12.687876 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 356/694] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Sep 25 04:23:12 2022] GET /identity/v3/role_assignments?user.id=23ea88f46e7740d295148a93ade29dd9&scope.project.id=056ca1b1a9974a7581550de0d9217823&role.id=9b44520f482c4423bd10568d0f86718c => generated 608 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:12.739570 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7869eec-5e53-4352-b7b1-faca39bbbac8 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:12.790523 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45ec054d-dd14-4415-8287-609607b3033f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:12.791169 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45ec054d-dd14-4415-8287-609607b3033f None admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:12.791449 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45ec054d-dd14-4415-8287-609607b3033f None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:12.791723 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45ec054d-dd14-4415-8287-609607b3033f None admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:12.792402 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45ec054d-dd14-4415-8287-609607b3033f None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:12.793992 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45ec054d-dd14-4415-8287-609607b3033f None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:12.832427 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 339/695] 10.222.0.25 () {60 vars in 1132 bytes} [Sun Sep 25 04:23:12 2022] POST /identity/v3/users => generated 254 bytes in 278 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 04:23:13.059841 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-721039f0-b0c8-4ada-8996-8c05069e7082 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:13.060339 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-721039f0-b0c8-4ada-8996-8c05069e7082 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:13.060339 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-721039f0-b0c8-4ada-8996-8c05069e7082 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:13.062687 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 357/696] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:23:13 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:23:13.068569 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74e364f0-b71e-478b-aa1a-8dfb6216f738 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:13.068950 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74e364f0-b71e-478b-aa1a-8dfb6216f738 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:13.069295 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74e364f0-b71e-478b-aa1a-8dfb6216f738 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:13.092852 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-74e364f0-b71e-478b-aa1a-8dfb6216f738 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:23:13.094832 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-74e364f0-b71e-478b-aa1a-8dfb6216f738 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:23:13.166034 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 340/697] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:23:13 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 04:23:13.458198 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-721039f0-b0c8-4ada-8996-8c05069e7082 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:13.471353 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:13.599838 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-721039f0-b0c8-4ada-8996-8c05069e7082 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:13.646744 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be14dff6-ff55-43b8-8186-d80ec2e0b7e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:13.647385 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be14dff6-ff55-43b8-8186-d80ec2e0b7e8 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:13.647663 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be14dff6-ff55-43b8-8186-d80ec2e0b7e8 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:13.647936 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be14dff6-ff55-43b8-8186-d80ec2e0b7e8 None admin] PATH_INFO: `/v3/services/network` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:13.648518 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be14dff6-ff55-43b8-8186-d80ec2e0b7e8 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:13.649906 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be14dff6-ff55-43b8-8186-d80ec2e0b7e8 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:13.663829 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-be14dff6-ff55-43b8-8186-d80ec2e0b7e8 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Sep 25 04:23:13.663829 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:13.663829 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 25 04:23:13.663829 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 25 04:23:13.663829 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 25 04:23:13.663829 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 25 04:23:13.663829 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 25 04:23:13.663829 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 04:23:13.663829 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 25 04:23:13.663829 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:23:13.663829 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 25 04:23:13.663829 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:23:13.663829 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:13.663829 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:13.663829 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:13.663829 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:13.663829 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:13.666375 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:13.666375 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:13.666375 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:13.666375 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:13.666375 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:13.666375 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:13.666375 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 25 04:23:13.666375 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 25 04:23:13.666375 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 25 04:23:13.666375 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 25 04:23:13.666375 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:13.666375 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:13.666375 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:13.666375 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:13.666375 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:13.666375 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:13.667451 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:13.667451 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:13.667451 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:13.667451 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:13.667451 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:13.667451 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:13.667451 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:13.667451 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:13.667451 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:13.667451 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:13.667451 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 25 04:23:13.667451 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 04:23:13.667451 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 25 04:23:13.667451 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:23:13.667451 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 358/698] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Sep 25 04:23:13 2022] GET /identity/v3/services/network => generated 88 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:13.673235 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74e364f0-b71e-478b-aa1a-8dfb6216f738 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:13.732836 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:13.828862 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74e364f0-b71e-478b-aa1a-8dfb6216f738 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:13.877279 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eff6bc95-a99c-4222-8e7b-6b90f070b1b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:13.877895 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eff6bc95-a99c-4222-8e7b-6b90f070b1b9 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:13.878171 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eff6bc95-a99c-4222-8e7b-6b90f070b1b9 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:13.878440 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eff6bc95-a99c-4222-8e7b-6b90f070b1b9 None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:13.879068 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eff6bc95-a99c-4222-8e7b-6b90f070b1b9 None admin] RBAC: Adding query filter params (name=network) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:13.879382 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eff6bc95-a99c-4222-8e7b-6b90f070b1b9 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:13.880778 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eff6bc95-a99c-4222-8e7b-6b90f070b1b9 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:13.885293 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 341/699] 10.222.0.25 () {56 vars in 1112 bytes} [Sun Sep 25 04:23:13 2022] GET /identity/v3/services?name=network => generated 124 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:13.891182 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be14dff6-ff55-43b8-8186-d80ec2e0b7e8 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:13.993368 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:14.026392 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be14dff6-ff55-43b8-8186-d80ec2e0b7e8 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:14.074444 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-208b57bb-c150-421f-886a-5cbdc6fefa5f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:14.075080 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-208b57bb-c150-421f-886a-5cbdc6fefa5f None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:14.075350 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-208b57bb-c150-421f-886a-5cbdc6fefa5f None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:14.075836 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-208b57bb-c150-421f-886a-5cbdc6fefa5f None admin] PATH_INFO: `/v3/services` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:14.076461 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-208b57bb-c150-421f-886a-5cbdc6fefa5f None admin] RBAC: Adding query filter params (type=network) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:14.076790 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-208b57bb-c150-421f-886a-5cbdc6fefa5f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:14.078718 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-208b57bb-c150-421f-886a-5cbdc6fefa5f None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:14.085230 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 359/700] 10.222.0.25 () {56 vars in 1112 bytes} [Sun Sep 25 04:23:13 2022] GET /identity/v3/services?type=network => generated 124 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:14.399263 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f90fb110-4ab9-4863-a37d-b7765850173b None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:14.399662 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f90fb110-4ab9-4863-a37d-b7765850173b None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:14.399865 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f90fb110-4ab9-4863-a37d-b7765850173b None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:14.401461 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 342/701] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:23:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:23:14.407031 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a08a61-a9e9-443a-a4ca-64e65c797999 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:14.407318 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a08a61-a9e9-443a-a4ca-64e65c797999 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:14.407593 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a08a61-a9e9-443a-a4ca-64e65c797999 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:14.427599 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-10a08a61-a9e9-443a-a4ca-64e65c797999 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:23:14.428804 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-10a08a61-a9e9-443a-a4ca-64e65c797999 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:23:14.508551 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 360/702] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:23:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 04:23:14.792028 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e344a583-4eb6-4626-9a1a-9e32b6c247f6 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:14.792532 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e344a583-4eb6-4626-9a1a-9e32b6c247f6 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:14.792532 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e344a583-4eb6-4626-9a1a-9e32b6c247f6 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:14.794095 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 343/703] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:23:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:23:14.804050 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1544a6c8-4fab-45a4-8ed5-3a5429f0bf1c None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:14.804337 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1544a6c8-4fab-45a4-8ed5-3a5429f0bf1c None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:14.804605 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1544a6c8-4fab-45a4-8ed5-3a5429f0bf1c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:14.833678 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-1544a6c8-4fab-45a4-8ed5-3a5429f0bf1c None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:23:14.834876 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-1544a6c8-4fab-45a4-8ed5-3a5429f0bf1c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:23:14.909967 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:14.918873 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 361/704] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:23:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 04:23:14.926650 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e344a583-4eb6-4626-9a1a-9e32b6c247f6 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:15.070923 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:15.077314 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e344a583-4eb6-4626-9a1a-9e32b6c247f6 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:15.131240 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37b17b07-3f95-415f-8d22-7449df1c005f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:15.131886 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b17b07-3f95-415f-8d22-7449df1c005f None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:15.132163 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b17b07-3f95-415f-8d22-7449df1c005f None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:15.132436 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b17b07-3f95-415f-8d22-7449df1c005f None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:15.148953 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37b17b07-3f95-415f-8d22-7449df1c005f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 04:23:15.153761 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-37b17b07-3f95-415f-8d22-7449df1c005f None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 04:23:15.153761 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:15.153761 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:15.153761 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:15.153761 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:15.153761 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:15.153761 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:15.153761 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:15.153761 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:15.153761 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:15.153761 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:15.153761 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:15.153761 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 04:23:15.153761 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 04:23:15.153761 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 04:23:15.153761 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 04:23:15.154753 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 04:23:15.154753 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 04:23:15.154753 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 04:23:15.154753 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 04:23:15.154753 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:15.154753 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:15.154753 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:15.154753 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:15.154753 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:15.154753 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:15.154753 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:15.154753 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:15.154753 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:15.154753 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:15.154753 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:15.154753 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:15.155741 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:15.155741 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:15.155741 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:15.155741 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:15.155741 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 04:23:15.155741 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 04:23:15.155741 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 04:23:15.155741 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 04:23:15.155741 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 04:23:15.155741 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 04:23:15.155741 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 04:23:15.155741 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:15.155741 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 344/705] 10.222.0.25 () {56 vars in 1096 bytes} [Sun Sep 25 04:23:14 2022] GET /identity/v3/roles/admin => generated 83 bytes in 229 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:15.162911 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1544a6c8-4fab-45a4-8ed5-3a5429f0bf1c None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:15.292447 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1544a6c8-4fab-45a4-8ed5-3a5429f0bf1c None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:15.340169 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6e5561e-abcd-4cdf-a81b-e293e6a0ea18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:15.340743 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6e5561e-abcd-4cdf-a81b-e293e6a0ea18 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:15.341025 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6e5561e-abcd-4cdf-a81b-e293e6a0ea18 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:15.341296 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6e5561e-abcd-4cdf-a81b-e293e6a0ea18 None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:15.341966 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6e5561e-abcd-4cdf-a81b-e293e6a0ea18 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:15.342270 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6e5561e-abcd-4cdf-a81b-e293e6a0ea18 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:15.344190 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6e5561e-abcd-4cdf-a81b-e293e6a0ea18 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:15.356745 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 362/706] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Sep 25 04:23:15 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:15.359458 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37b17b07-3f95-415f-8d22-7449df1c005f None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:15.414468 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e65ffb-6f24-48ac-b989-60cedf00bfd4 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:15.414795 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e65ffb-6f24-48ac-b989-60cedf00bfd4 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:15.415024 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e65ffb-6f24-48ac-b989-60cedf00bfd4 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:15.416351 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 363/707] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:23:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 25 04:23:15.422232 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5419754-8672-4fd0-a4ea-9ad24127419e None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:15.422708 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5419754-8672-4fd0-a4ea-9ad24127419e None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:15.422984 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5419754-8672-4fd0-a4ea-9ad24127419e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:15.441972 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-d5419754-8672-4fd0-a4ea-9ad24127419e None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:23:15.443159 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-d5419754-8672-4fd0-a4ea-9ad24127419e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:23:15.499025 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37b17b07-3f95-415f-8d22-7449df1c005f None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:15.518040 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:15.534091 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 364/708] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:23:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 25 04:23:15.535316 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5419754-8672-4fd0-a4ea-9ad24127419e None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:15.544013 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6364f3d-0d42-472d-bcc3-a03721eaa273 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:15.544686 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6364f3d-0d42-472d-bcc3-a03721eaa273 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:15.544973 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6364f3d-0d42-472d-bcc3-a03721eaa273 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:15.545245 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6364f3d-0d42-472d-bcc3-a03721eaa273 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:15.568388 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6364f3d-0d42-472d-bcc3-a03721eaa273 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:15.569810 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6364f3d-0d42-472d-bcc3-a03721eaa273 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:15.600484 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-e6364f3d-0d42-472d-bcc3-a03721eaa273 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:23:15.600484 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:15.600484 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:15.600484 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:15.600484 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:15.600484 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:15.600484 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:15.600484 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:15.600484 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:15.600484 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:15.600484 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:15.600484 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:15.600484 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 04:23:15.600484 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 04:23:15.600484 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 04:23:15.600484 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 04:23:15.601512 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:15.601512 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:15.601512 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:15.601512 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:15.601512 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:15.601512 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:15.601512 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:15.601512 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:15.601512 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:15.601512 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:15.601512 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:15.601512 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:15.601512 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:15.601512 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:15.601512 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:15.601512 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:15.601512 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 04:23:15.603038 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 04:23:15.603038 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:23:15.603038 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:15.603038 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 345/709] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:23:15 2022] GET /identity/v3/domains/Default => generated 87 bytes in 243 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:15.607105 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6364f3d-0d42-472d-bcc3-a03721eaa273 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:15.627404 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:15.691067 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5419754-8672-4fd0-a4ea-9ad24127419e None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:15.753409 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffbd9bdc-6481-417c-8696-037218904b54 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:15.754051 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffbd9bdc-6481-417c-8696-037218904b54 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:15.754335 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffbd9bdc-6481-417c-8696-037218904b54 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:15.754608 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffbd9bdc-6481-417c-8696-037218904b54 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:15.755198 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffbd9bdc-6481-417c-8696-037218904b54 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:15.756568 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffbd9bdc-6481-417c-8696-037218904b54 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:15.773285 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-ffbd9bdc-6481-417c-8696-037218904b54 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 25 04:23:15.773285 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:15.773285 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 25 04:23:15.773285 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 25 04:23:15.773285 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 25 04:23:15.773285 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 25 04:23:15.773285 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 25 04:23:15.773285 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 04:23:15.773285 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 25 04:23:15.773285 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:23:15.773285 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 25 04:23:15.773285 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:23:15.773285 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:15.773285 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:15.773285 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:15.773285 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:15.773285 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:15.774344 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:15.774344 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:15.774344 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:15.774344 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:15.774344 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:15.774344 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:15.774344 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 25 04:23:15.774344 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 25 04:23:15.774344 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 25 04:23:15.774344 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 25 04:23:15.774344 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:15.774344 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:15.774344 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:15.774344 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:15.774344 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:15.774344 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:15.775571 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:15.775571 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:15.775571 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:15.775571 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:15.775571 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:15.775571 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:15.775571 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:15.775571 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:15.775571 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:15.775571 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:15.775571 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 25 04:23:15.775571 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 04:23:15.775571 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 25 04:23:15.775571 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:23:15.775571 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 365/710] 10.222.0.25 () {56 vars in 1129 bytes} [Sun Sep 25 04:23:15 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 239 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:15.785930 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffbd9bdc-6481-417c-8696-037218904b54 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:15.806261 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6364f3d-0d42-472d-bcc3-a03721eaa273 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:15.853429 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec9116e4-9c8f-44bc-9aa4-18f0212695e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:15.853900 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec9116e4-9c8f-44bc-9aa4-18f0212695e8 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:15.854512 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec9116e4-9c8f-44bc-9aa4-18f0212695e8 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:15.854795 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec9116e4-9c8f-44bc-9aa4-18f0212695e8 None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:15.855591 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec9116e4-9c8f-44bc-9aa4-18f0212695e8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:15.855903 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec9116e4-9c8f-44bc-9aa4-18f0212695e8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:15.857315 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec9116e4-9c8f-44bc-9aa4-18f0212695e8 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:15.867735 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:15.870640 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 346/711] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Sep 25 04:23:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 264 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:23:15.872041 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907dd8b9-1077-4dbb-918a-7caaaf87a5dd None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:15.872527 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907dd8b9-1077-4dbb-918a-7caaaf87a5dd None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:15.872956 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907dd8b9-1077-4dbb-918a-7caaaf87a5dd None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:15.874342 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 347/712] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:23:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:23:15.875242 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-907dd8b9-1077-4dbb-918a-7caaaf87a5dd None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:15.924197 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffbd9bdc-6481-417c-8696-037218904b54 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:15.955288 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:15.959247 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9d1579f-9c80-49e8-8857-b9ae3ceaab50 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:15.959866 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9d1579f-9c80-49e8-8857-b9ae3ceaab50 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:15.960146 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9d1579f-9c80-49e8-8857-b9ae3ceaab50 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:15.960416 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9d1579f-9c80-49e8-8857-b9ae3ceaab50 None admin] PATH_INFO: `/v3/services` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:15.961066 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9d1579f-9c80-49e8-8857-b9ae3ceaab50 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:15.961369 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9d1579f-9c80-49e8-8857-b9ae3ceaab50 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:15.962729 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9d1579f-9c80-49e8-8857-b9ae3ceaab50 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:15.967130 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 366/713] 10.222.0.25 () {56 vars in 1124 bytes} [Sun Sep 25 04:23:15 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:15.972680 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9d1579f-9c80-49e8-8857-b9ae3ceaab50 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:15.990991 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-907dd8b9-1077-4dbb-918a-7caaaf87a5dd None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:16.027415 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6be88822-9e38-44df-ac0e-d5a47b121d02 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:16.028075 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6be88822-9e38-44df-ac0e-d5a47b121d02 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:16.028358 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6be88822-9e38-44df-ac0e-d5a47b121d02 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:16.028625 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6be88822-9e38-44df-ac0e-d5a47b121d02 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:16.040351 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6be88822-9e38-44df-ac0e-d5a47b121d02 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 04:23:16.045346 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-6be88822-9e38-44df-ac0e-d5a47b121d02 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 04:23:16.045346 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:16.045346 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:16.045346 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:16.045346 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:16.045346 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:16.045346 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:16.045346 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:16.045346 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:16.045346 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:16.045346 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:16.045346 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:16.045346 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 04:23:16.045346 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 04:23:16.045346 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 04:23:16.045346 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 04:23:16.046355 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 04:23:16.046355 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 04:23:16.046355 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 04:23:16.046355 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 04:23:16.046355 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:16.046355 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:16.046355 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:16.046355 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:16.046355 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:16.046355 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:16.046355 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:16.046355 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:16.046355 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:16.046355 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:16.046355 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:16.046355 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:16.047255 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:16.047255 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:16.047255 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:16.047255 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:16.047255 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 04:23:16.047255 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 04:23:16.047255 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 04:23:16.047255 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 04:23:16.047255 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 04:23:16.047255 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 04:23:16.047255 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 04:23:16.047255 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:16.047255 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 348/714] 10.222.0.25 () {56 vars in 1096 bytes} [Sun Sep 25 04:23:15 2022] GET /identity/v3/roles/admin => generated 83 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:16.048430 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4a01fdf-e2b2-45cb-afa1-02b5788144bf None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:16.048430 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4a01fdf-e2b2-45cb-afa1-02b5788144bf None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:16.048594 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4a01fdf-e2b2-45cb-afa1-02b5788144bf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:16.067448 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-b4a01fdf-e2b2-45cb-afa1-02b5788144bf None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:23:16.068904 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-b4a01fdf-e2b2-45cb-afa1-02b5788144bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:23:16.077063 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9d1579f-9c80-49e8-8857-b9ae3ceaab50 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:16.095552 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:16.113380 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09e1ae29-ff22-4fb1-9d61-aa09f6796cfd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:16.114364 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09e1ae29-ff22-4fb1-9d61-aa09f6796cfd None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:16.114648 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09e1ae29-ff22-4fb1-9d61-aa09f6796cfd None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:16.114920 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09e1ae29-ff22-4fb1-9d61-aa09f6796cfd None admin] PATH_INFO: `/v3/users/nova` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:16.125598 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09e1ae29-ff22-4fb1-9d61-aa09f6796cfd None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:16.127190 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09e1ae29-ff22-4fb1-9d61-aa09f6796cfd None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:16.139081 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-09e1ae29-ff22-4fb1-9d61-aa09f6796cfd None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 25 04:23:16.139081 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:16.139081 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:16.139081 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:16.139081 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:16.139081 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:16.139081 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:16.139081 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:16.139081 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:16.139081 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:16.139081 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:16.139081 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:16.139081 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 25 04:23:16.139081 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 25 04:23:16.139081 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 25 04:23:16.139081 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 25 04:23:16.140382 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:16.140382 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:16.140382 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 25 04:23:16.140382 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 04:23:16.140382 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 25 04:23:16.140382 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 04:23:16.140382 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:16.140382 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:16.140382 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:16.140382 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:16.140382 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:16.140382 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:16.140382 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:16.140382 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:16.140382 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:16.140382 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:16.140382 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:16.140382 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:16.141468 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:16.141468 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:16.141468 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 25 04:23:16.141468 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 25 04:23:16.141468 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 25 04:23:16.141468 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 25 04:23:16.141468 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 25 04:23:16.141468 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 25 04:23:16.141468 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 25 04:23:16.141468 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:23:16.142024 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 349/715] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:23:16 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 04:23:16.142024 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4a01fdf-e2b2-45cb-afa1-02b5788144bf None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:16.153826 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 367/716] 10.222.0.25 () {56 vars in 1093 bytes} [Sun Sep 25 04:23:15 2022] GET /identity/v3/users/nova => generated 82 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:16.153826 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09e1ae29-ff22-4fb1-9d61-aa09f6796cfd None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:16.208775 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:16.255553 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4a01fdf-e2b2-45cb-afa1-02b5788144bf None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:16.286426 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09e1ae29-ff22-4fb1-9d61-aa09f6796cfd None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:16.301226 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c3e28d2-8283-4067-8bc9-b9b204719a84 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:16.301844 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c3e28d2-8283-4067-8bc9-b9b204719a84 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:16.302122 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c3e28d2-8283-4067-8bc9-b9b204719a84 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:16.302389 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c3e28d2-8283-4067-8bc9-b9b204719a84 None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:16.303019 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c3e28d2-8283-4067-8bc9-b9b204719a84 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:16.303323 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c3e28d2-8283-4067-8bc9-b9b204719a84 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:16.304808 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c3e28d2-8283-4067-8bc9-b9b204719a84 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:16.316979 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 350/717] 10.222.0.25 () {56 vars in 1124 bytes} [Sun Sep 25 04:23:16 2022] GET /identity/v3/services?type=block-storage => generated 130 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:16.317944 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c3e28d2-8283-4067-8bc9-b9b204719a84 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:16.329868 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac70bd39-704b-476e-a6a2-ded31d43c5d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:16.330864 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac70bd39-704b-476e-a6a2-ded31d43c5d5 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:16.331341 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac70bd39-704b-476e-a6a2-ded31d43c5d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:16.331806 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac70bd39-704b-476e-a6a2-ded31d43c5d5 None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:16.333281 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac70bd39-704b-476e-a6a2-ded31d43c5d5 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:16.333805 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac70bd39-704b-476e-a6a2-ded31d43c5d5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:16.335443 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac70bd39-704b-476e-a6a2-ded31d43c5d5 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:16.347907 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 368/718] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Sep 25 04:23:16 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:16.355563 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac70bd39-704b-476e-a6a2-ded31d43c5d5 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:16.445029 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:16.481465 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c3e28d2-8283-4067-8bc9-b9b204719a84 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:16.513406 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:16.516593 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac70bd39-704b-476e-a6a2-ded31d43c5d5 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:16.527367 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbf83825-1b2f-4ad0-a7cd-675c467770b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:16.528336 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbf83825-1b2f-4ad0-a7cd-675c467770b5 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:16.528787 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbf83825-1b2f-4ad0-a7cd-675c467770b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:16.529230 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbf83825-1b2f-4ad0-a7cd-675c467770b5 None admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:16.530208 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbf83825-1b2f-4ad0-a7cd-675c467770b5 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:16.530767 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbf83825-1b2f-4ad0-a7cd-675c467770b5 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:16.532370 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbf83825-1b2f-4ad0-a7cd-675c467770b5 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:16.552772 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 351/719] 10.222.0.25 () {56 vars in 1133 bytes} [Sun Sep 25 04:23:16 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 235 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:16.554182 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbf83825-1b2f-4ad0-a7cd-675c467770b5 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:16.556406 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4645acea-3a0e-482a-86df-d4918bfa1be1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:16.557475 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4645acea-3a0e-482a-86df-d4918bfa1be1 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:16.557863 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4645acea-3a0e-482a-86df-d4918bfa1be1 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:16.558258 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4645acea-3a0e-482a-86df-d4918bfa1be1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:16.575953 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4645acea-3a0e-482a-86df-d4918bfa1be1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:16.579182 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4645acea-3a0e-482a-86df-d4918bfa1be1 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:16.595640 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-4645acea-3a0e-482a-86df-d4918bfa1be1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:23:16.595640 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:16.595640 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:16.595640 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:16.595640 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:16.595640 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:16.595640 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:16.595640 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:16.595640 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:16.595640 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:16.595640 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:16.595640 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:16.595640 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 04:23:16.595640 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 04:23:16.595640 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 04:23:16.595640 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 04:23:16.596698 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:16.596698 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:16.596698 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:16.596698 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:16.596698 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:16.596698 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:16.596698 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:16.596698 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:16.596698 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:16.596698 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:16.596698 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:16.596698 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:16.596698 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:16.596698 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:16.596698 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:16.596698 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:16.596698 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 04:23:16.597644 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 04:23:16.597644 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:23:16.597644 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:23:16.601281 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 369/720] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:23:16 2022] GET /identity/v3/domains/Default => generated 87 bytes in 242 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 04:23:16.603647 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4645acea-3a0e-482a-86df-d4918bfa1be1 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:16.688387 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbf83825-1b2f-4ad0-a7cd-675c467770b5 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:16.699319 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:16.710411 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4645acea-3a0e-482a-86df-d4918bfa1be1 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:16.724688 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9397630-90ab-4638-ad95-8a4c776e0f01 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:16.725333 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9397630-90ab-4638-ad95-8a4c776e0f01 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:16.725617 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9397630-90ab-4638-ad95-8a4c776e0f01 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:16.725883 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9397630-90ab-4638-ad95-8a4c776e0f01 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:16.737500 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9397630-90ab-4638-ad95-8a4c776e0f01 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 04:23:16.742287 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-d9397630-90ab-4638-ad95-8a4c776e0f01 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 25 04:23:16.742287 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:16.742287 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:16.742287 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:16.742287 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:16.742287 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:16.742287 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:16.742287 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:16.742287 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:16.742287 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:16.742287 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:16.742287 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:16.742287 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 04:23:16.742287 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 04:23:16.742287 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 04:23:16.742287 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 04:23:16.743437 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 04:23:16.743437 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 04:23:16.743437 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 04:23:16.743437 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 04:23:16.743437 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:16.743437 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:16.743437 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:16.743437 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:16.743437 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:16.743437 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:16.743437 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:16.743437 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:16.743437 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:16.743437 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:16.743437 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:16.743437 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:16.744296 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:16.744296 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:16.744296 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:16.744296 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:16.744296 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 04:23:16.744296 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 04:23:16.744296 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 04:23:16.744296 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 04:23:16.744296 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 04:23:16.744296 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 04:23:16.744296 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 25 04:23:16.744296 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:16.744296 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 352/721] 10.222.0.25 () {56 vars in 1102 bytes} [Sun Sep 25 04:23:16 2022] GET /identity/v3/roles/service => generated 85 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:16.745036 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9397630-90ab-4638-ad95-8a4c776e0f01 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:16.753387 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9f5669d-e772-478a-adc2-c2dadd80c556 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:16.754014 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9f5669d-e772-478a-adc2-c2dadd80c556 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:16.754298 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9f5669d-e772-478a-adc2-c2dadd80c556 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:16.754574 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9f5669d-e772-478a-adc2-c2dadd80c556 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:16.794203 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:16.829661 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9f5669d-e772-478a-adc2-c2dadd80c556 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:16.831226 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9f5669d-e772-478a-adc2-c2dadd80c556 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:16.860426 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9397630-90ab-4638-ad95-8a4c776e0f01 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:16.895217 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-784f30a3-b16b-4be4-b4ca-b6fe5cf5a3a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:16.895837 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-784f30a3-b16b-4be4-b4ca-b6fe5cf5a3a7 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:16.896229 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-784f30a3-b16b-4be4-b4ca-b6fe5cf5a3a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:16.896560 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-784f30a3-b16b-4be4-b4ca-b6fe5cf5a3a7 None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:16.897235 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-784f30a3-b16b-4be4-b4ca-b6fe5cf5a3a7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:16.897572 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-784f30a3-b16b-4be4-b4ca-b6fe5cf5a3a7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:16.899079 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-784f30a3-b16b-4be4-b4ca-b6fe5cf5a3a7 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:16.916529 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 353/722] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Sep 25 04:23:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:16.917851 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e238bdb-058c-44aa-afcf-30f2c94518d7 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:16.918141 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e238bdb-058c-44aa-afcf-30f2c94518d7 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:16.918418 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e238bdb-058c-44aa-afcf-30f2c94518d7 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:16.920676 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 354/723] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:23:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:23:16.920676 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e238bdb-058c-44aa-afcf-30f2c94518d7 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:16.937685 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:16.947847 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 370/724] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Sep 25 04:23:16 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 347 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 25 04:23:16.948851 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9f5669d-e772-478a-adc2-c2dadd80c556 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:17.044258 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e238bdb-058c-44aa-afcf-30f2c94518d7 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:17.046642 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:17.053077 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9f5669d-e772-478a-adc2-c2dadd80c556 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:17.083817 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d21e7baf-7daa-439a-b053-b6b6f661b3e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:17.084457 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d21e7baf-7daa-439a-b053-b6b6f661b3e2 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:17.084751 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d21e7baf-7daa-439a-b053-b6b6f661b3e2 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:17.085026 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d21e7baf-7daa-439a-b053-b6b6f661b3e2 None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:17.085688 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d21e7baf-7daa-439a-b053-b6b6f661b3e2 None admin] RBAC: Adding query filter params (name=service) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:17.085993 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d21e7baf-7daa-439a-b053-b6b6f661b3e2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:17.087436 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d21e7baf-7daa-439a-b053-b6b6f661b3e2 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:17.090145 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b8194f2-4b4b-438a-864c-ef7b497e4aa7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:17.090790 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b8194f2-4b4b-438a-864c-ef7b497e4aa7 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:17.091071 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b8194f2-4b4b-438a-864c-ef7b497e4aa7 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:17.091343 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b8194f2-4b4b-438a-864c-ef7b497e4aa7 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:17.100619 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 355/725] 10.222.0.25 () {56 vars in 1103 bytes} [Sun Sep 25 04:23:16 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:17.100619 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c9d51e7-51d9-4a89-bce9-516478ebc6fd None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:17.100619 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c9d51e7-51d9-4a89-bce9-516478ebc6fd None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:17.100619 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c9d51e7-51d9-4a89-bce9-516478ebc6fd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:17.102392 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b8194f2-4b4b-438a-864c-ef7b497e4aa7 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:17.103851 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b8194f2-4b4b-438a-864c-ef7b497e4aa7 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:17.117242 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-4b8194f2-4b4b-438a-864c-ef7b497e4aa7 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 25 04:23:17.117242 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:17.117242 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:17.117242 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:17.117242 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:17.117242 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:17.117242 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:17.117242 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:17.117242 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:17.117242 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:17.117242 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:17.117242 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:17.117242 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 25 04:23:17.117242 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 25 04:23:17.117242 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 25 04:23:17.117242 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 25 04:23:17.118251 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:17.118251 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:17.118251 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 25 04:23:17.118251 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 04:23:17.118251 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 25 04:23:17.118251 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 04:23:17.118251 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:17.118251 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:17.118251 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:17.118251 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:17.118251 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:17.118251 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:17.118251 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:17.118251 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:17.118251 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:17.118251 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:17.118251 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:17.118251 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:17.119255 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:17.119255 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:17.119255 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 25 04:23:17.119255 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 25 04:23:17.119255 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 25 04:23:17.119255 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 25 04:23:17.119255 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 25 04:23:17.119255 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 25 04:23:17.119255 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 25 04:23:17.119255 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:23:17.119834 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-9c9d51e7-51d9-4a89-bce9-516478ebc6fd None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:23:17.119921 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 371/726] 10.222.0.25 () {56 vars in 1096 bytes} [Sun Sep 25 04:23:16 2022] GET /identity/v3/users/swift => generated 83 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:17.119921 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b8194f2-4b4b-438a-864c-ef7b497e4aa7 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:17.123590 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-9c9d51e7-51d9-4a89-bce9-516478ebc6fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:23:17.177611 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:17.204681 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 356/727] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:23:17 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 04:23:17.204681 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c9d51e7-51d9-4a89-bce9-516478ebc6fd None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:17.236534 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b8194f2-4b4b-438a-864c-ef7b497e4aa7 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:17.272127 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f094f4f8-ec87-493c-a92e-f6ff17d2f45d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:17.272807 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f094f4f8-ec87-493c-a92e-f6ff17d2f45d None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:17.273091 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f094f4f8-ec87-493c-a92e-f6ff17d2f45d None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:17.273364 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f094f4f8-ec87-493c-a92e-f6ff17d2f45d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:17.285206 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f094f4f8-ec87-493c-a92e-f6ff17d2f45d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:17.286520 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f094f4f8-ec87-493c-a92e-f6ff17d2f45d None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:17.302822 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-f094f4f8-ec87-493c-a92e-f6ff17d2f45d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:23:17.302822 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:17.302822 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:17.302822 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:17.302822 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:17.302822 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:17.302822 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:17.302822 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:17.302822 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:17.302822 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:17.302822 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:17.302822 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:17.302822 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 04:23:17.302822 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 04:23:17.302822 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 04:23:17.302822 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 04:23:17.304037 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:17.304037 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:17.304037 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:17.304037 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:17.304037 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:17.304037 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:17.304037 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:17.304037 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:17.304037 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:17.304037 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:17.304037 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:17.304037 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:17.304037 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:17.304037 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:17.304037 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:17.304037 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:17.304037 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 04:23:17.305174 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 04:23:17.305174 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:23:17.305174 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:23:17.311437 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 372/728] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:23:17 2022] GET /identity/v3/domains/Default => generated 87 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:17.312567 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f094f4f8-ec87-493c-a92e-f6ff17d2f45d None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:17.332698 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:17.343671 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c9d51e7-51d9-4a89-bce9-516478ebc6fd None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:17.381875 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3f2f0dc-44fc-49bf-b23d-0d2af086906d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:17.382518 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3f2f0dc-44fc-49bf-b23d-0d2af086906d None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:17.382799 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3f2f0dc-44fc-49bf-b23d-0d2af086906d None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:17.383074 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3f2f0dc-44fc-49bf-b23d-0d2af086906d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:17.395046 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3f2f0dc-44fc-49bf-b23d-0d2af086906d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:17.396655 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3f2f0dc-44fc-49bf-b23d-0d2af086906d None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:17.412325 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-a3f2f0dc-44fc-49bf-b23d-0d2af086906d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:23:17.412325 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:17.412325 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:17.412325 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:17.412325 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:17.412325 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:17.412325 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:17.412325 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:17.412325 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:17.412325 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:17.412325 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:17.412325 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:17.412325 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 04:23:17.412325 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 04:23:17.412325 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 04:23:17.412325 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 04:23:17.413359 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:17.413359 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:17.413359 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:17.413359 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:17.413359 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:17.413359 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:17.413359 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:17.413359 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:17.413359 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:17.413359 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:17.413359 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:17.413359 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:17.413359 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:17.413359 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:17.413359 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:17.413359 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:17.413359 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 04:23:17.414290 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 04:23:17.414290 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:23:17.414290 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:17.417006 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 357/729] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:23:17 2022] GET /identity/v3/domains/Default => generated 87 bytes in 219 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:17.417886 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3f2f0dc-44fc-49bf-b23d-0d2af086906d None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:17.447635 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f094f4f8-ec87-493c-a92e-f6ff17d2f45d None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:17.451518 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:17.495652 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba3cf722-844d-4a6a-b543-56dd87cb5dea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:17.496350 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba3cf722-844d-4a6a-b543-56dd87cb5dea None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:17.496649 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba3cf722-844d-4a6a-b543-56dd87cb5dea None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:17.496952 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba3cf722-844d-4a6a-b543-56dd87cb5dea None admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:17.497680 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba3cf722-844d-4a6a-b543-56dd87cb5dea None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:17.497991 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba3cf722-844d-4a6a-b543-56dd87cb5dea None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:17.500609 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba3cf722-844d-4a6a-b543-56dd87cb5dea None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:17.517754 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 373/730] 10.222.0.25 () {56 vars in 1135 bytes} [Sun Sep 25 04:23:17 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:17.523213 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba3cf722-844d-4a6a-b543-56dd87cb5dea None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:17.534440 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3f2f0dc-44fc-49bf-b23d-0d2af086906d None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:17.570828 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25eae812-f2b3-45b7-a356-ca8be332824a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:17.571366 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25eae812-f2b3-45b7-a356-ca8be332824a None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:17.571654 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25eae812-f2b3-45b7-a356-ca8be332824a None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:17.571931 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25eae812-f2b3-45b7-a356-ca8be332824a None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:17.572571 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25eae812-f2b3-45b7-a356-ca8be332824a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:17.572910 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25eae812-f2b3-45b7-a356-ca8be332824a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:17.574285 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25eae812-f2b3-45b7-a356-ca8be332824a None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:17.587609 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 358/731] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Sep 25 04:23:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:17.590174 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25eae812-f2b3-45b7-a356-ca8be332824a None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:17.599865 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:17.627810 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba3cf722-844d-4a6a-b543-56dd87cb5dea None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:17.664836 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f05ebd9c-50c6-4bdb-96ae-939868ce920a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:17.665451 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f05ebd9c-50c6-4bdb-96ae-939868ce920a None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:17.665733 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f05ebd9c-50c6-4bdb-96ae-939868ce920a None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:17.666004 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f05ebd9c-50c6-4bdb-96ae-939868ce920a None admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:17.666642 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f05ebd9c-50c6-4bdb-96ae-939868ce920a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:17.666946 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f05ebd9c-50c6-4bdb-96ae-939868ce920a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:17.668400 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f05ebd9c-50c6-4bdb-96ae-939868ce920a None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:17.685524 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 374/732] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Sep 25 04:23:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:17.685524 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f05ebd9c-50c6-4bdb-96ae-939868ce920a None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:17.707465 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:17.708398 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25eae812-f2b3-45b7-a356-ca8be332824a None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:17.747142 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a816ee37-3bcb-496b-b455-3b416dd09027 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:17.747749 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a816ee37-3bcb-496b-b455-3b416dd09027 None admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:17.748027 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a816ee37-3bcb-496b-b455-3b416dd09027 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:17.748294 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a816ee37-3bcb-496b-b455-3b416dd09027 None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:17.748992 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a816ee37-3bcb-496b-b455-3b416dd09027 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:17.751153 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a816ee37-3bcb-496b-b455-3b416dd09027 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:17.766719 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 359/733] 10.222.0.25 () {60 vars in 1141 bytes} [Sun Sep 25 04:23:17 2022] POST /identity/v3/services => generated 240 bytes in 179 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:23:17.767844 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a816ee37-3bcb-496b-b455-3b416dd09027 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:17.800268 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f05ebd9c-50c6-4bdb-96ae-939868ce920a None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:17.863384 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:17.866087 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ead8969-b75a-43c0-afb3-e2cc46468b65 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:17.867071 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ead8969-b75a-43c0-afb3-e2cc46468b65 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:17.867560 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ead8969-b75a-43c0-afb3-e2cc46468b65 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:17.868061 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ead8969-b75a-43c0-afb3-e2cc46468b65 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:17.906120 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a816ee37-3bcb-496b-b455-3b416dd09027 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:17.944360 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48bc70b3-dbd6-4fe5-ab29-d364131a7e17 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:17.945986 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48bc70b3-dbd6-4fe5-ab29-d364131a7e17 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:17.945986 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48bc70b3-dbd6-4fe5-ab29-d364131a7e17 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:17.946188 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48bc70b3-dbd6-4fe5-ab29-d364131a7e17 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:17.955060 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48bc70b3-dbd6-4fe5-ab29-d364131a7e17 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:17.955767 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48bc70b3-dbd6-4fe5-ab29-d364131a7e17 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:17.956884 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ead8969-b75a-43c0-afb3-e2cc46468b65 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:17.958528 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ead8969-b75a-43c0-afb3-e2cc46468b65 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:17.972145 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-48bc70b3-dbd6-4fe5-ab29-d364131a7e17 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 04:23:17.972145 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:17.972145 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:17.972145 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:17.972145 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:17.972145 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:17.972145 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:17.972145 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:17.972145 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:17.972145 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:17.972145 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:17.972145 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:17.972145 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 25 04:23:17.972145 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 25 04:23:17.972145 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 25 04:23:17.973125 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 25 04:23:17.973125 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:17.973125 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:17.973125 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:17.973125 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:17.973125 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:17.973125 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:17.973125 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:17.973125 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:17.973125 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:17.973125 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:17.973125 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:17.973125 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:17.973125 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:17.973125 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:17.973125 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:17.973125 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:17.974040 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 25 04:23:17.974040 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 25 04:23:17.974040 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 25 04:23:17.974040 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 25 04:23:17.974040 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 25 04:23:17.974040 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 25 04:23:17.974040 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 04:23:17.974040 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:17.978605 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 360/734] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Sep 25 04:23:17 2022] GET /identity/v3/projects/service => generated 88 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:17.979492 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48bc70b3-dbd6-4fe5-ab29-d364131a7e17 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:17.983369 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:18.087955 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 375/735] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Sep 25 04:23:17 2022] GET /identity/v3/auth/tokens => generated 898 bytes in 406 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 25 04:23:18.089218 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ead8969-b75a-43c0-afb3-e2cc46468b65 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:18.110489 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48bc70b3-dbd6-4fe5-ab29-d364131a7e17 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:18.141482 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:18.151716 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5b6285d-435f-4014-9350-385bd3fef947 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:18.152502 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5b6285d-435f-4014-9350-385bd3fef947 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:18.152906 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5b6285d-435f-4014-9350-385bd3fef947 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:18.153333 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5b6285d-435f-4014-9350-385bd3fef947 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:18.162053 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5b6285d-435f-4014-9350-385bd3fef947 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:18.163910 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5b6285d-435f-4014-9350-385bd3fef947 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:18.176528 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-b5b6285d-435f-4014-9350-385bd3fef947 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 25 04:23:18.176528 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:18.176528 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:18.176528 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:18.176528 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:18.176528 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:18.176528 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:18.176528 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:18.176528 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:18.176528 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:18.176528 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:18.176528 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:18.176528 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 25 04:23:18.176528 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 25 04:23:18.176528 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 25 04:23:18.176528 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 25 04:23:18.177563 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:18.177563 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:18.177563 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 25 04:23:18.177563 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 04:23:18.177563 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 25 04:23:18.177563 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 04:23:18.177563 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:18.177563 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:18.177563 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:18.177563 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:18.177563 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:18.177563 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:18.177563 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:18.177563 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:18.177563 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:18.177563 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:18.177563 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:18.177563 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:18.178516 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:18.178516 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:18.178516 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 25 04:23:18.178516 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 25 04:23:18.178516 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 25 04:23:18.178516 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 25 04:23:18.178516 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 25 04:23:18.178516 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 25 04:23:18.178516 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 25 04:23:18.178516 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:18.182602 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 361/736] 10.222.0.25 () {56 vars in 1117 bytes} [Sun Sep 25 04:23:17 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:18.183354 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5b6285d-435f-4014-9350-385bd3fef947 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:18.220204 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ead8969-b75a-43c0-afb3-e2cc46468b65 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:18.242352 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:18.263830 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97ddf7ec-22c0-4dd2-ae0c-be08840ca799 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:18.264450 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ddf7ec-22c0-4dd2-ae0c-be08840ca799 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:18.264760 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ddf7ec-22c0-4dd2-ae0c-be08840ca799 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:18.265103 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ddf7ec-22c0-4dd2-ae0c-be08840ca799 None admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:18.266020 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97ddf7ec-22c0-4dd2-ae0c-be08840ca799 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:18.266333 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97ddf7ec-22c0-4dd2-ae0c-be08840ca799 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:18.267862 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97ddf7ec-22c0-4dd2-ae0c-be08840ca799 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:18.282763 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 376/737] 10.222.0.25 () {56 vars in 1148 bytes} [Sun Sep 25 04:23:18 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:18.283609 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97ddf7ec-22c0-4dd2-ae0c-be08840ca799 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:18.295885 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5b6285d-435f-4014-9350-385bd3fef947 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:18.333019 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-173b50af-255e-4fde-9826-461328a1518e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:18.333662 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-173b50af-255e-4fde-9826-461328a1518e None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:18.333941 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-173b50af-255e-4fde-9826-461328a1518e None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:18.334209 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-173b50af-255e-4fde-9826-461328a1518e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:18.346176 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-173b50af-255e-4fde-9826-461328a1518e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:18.347635 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-173b50af-255e-4fde-9826-461328a1518e None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:18.362389 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-173b50af-255e-4fde-9826-461328a1518e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:23:18.362389 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:18.362389 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:18.362389 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:18.362389 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:18.362389 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:18.362389 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:18.362389 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:18.362389 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:18.362389 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:18.362389 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:18.362389 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:18.362389 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 04:23:18.362389 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 04:23:18.362389 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 04:23:18.362389 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 04:23:18.364534 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:18.364534 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:18.364534 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:18.364534 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:18.364534 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:18.364534 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:18.364534 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:18.364534 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:18.364534 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:18.364534 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:18.364534 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:18.364534 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:18.364534 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:18.364534 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:18.364534 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:18.364534 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:18.364534 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 04:23:18.366743 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 04:23:18.366743 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:23:18.366743 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:18.366743 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 362/738] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:23:18 2022] GET /identity/v3/domains/Default => generated 87 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:18.366743 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-173b50af-255e-4fde-9826-461328a1518e None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:18.407061 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97ddf7ec-22c0-4dd2-ae0c-be08840ca799 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:18.408206 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:18.449798 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97e19160-aae1-444a-bd4c-81e191dc7d70 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:18.451077 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e19160-aae1-444a-bd4c-81e191dc7d70 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:18.451634 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e19160-aae1-444a-bd4c-81e191dc7d70 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:18.452009 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e19160-aae1-444a-bd4c-81e191dc7d70 None admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:18.452849 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97e19160-aae1-444a-bd4c-81e191dc7d70 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:18.453247 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97e19160-aae1-444a-bd4c-81e191dc7d70 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:18.454811 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97e19160-aae1-444a-bd4c-81e191dc7d70 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:18.481444 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-173b50af-255e-4fde-9826-461328a1518e None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:18.490414 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 377/739] 10.222.0.25 () {56 vars in 1149 bytes} [Sun Sep 25 04:23:18 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:18.491539 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97e19160-aae1-444a-bd4c-81e191dc7d70 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:18.533428 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae3c6679-2c41-43fc-8c8c-af21ed3abd92 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:18.534969 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:18.536175 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae3c6679-2c41-43fc-8c8c-af21ed3abd92 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:18.536685 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae3c6679-2c41-43fc-8c8c-af21ed3abd92 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:18.537315 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae3c6679-2c41-43fc-8c8c-af21ed3abd92 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:18.538357 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae3c6679-2c41-43fc-8c8c-af21ed3abd92 None admin] RBAC: Adding query filter params (role.id=c51fe7c5ad804040879acfb54269455f, scope.project.id=056ca1b1a9974a7581550de0d9217823, user.id=e703b42c53974daaa608b0ccc3f018b1) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:18.538874 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae3c6679-2c41-43fc-8c8c-af21ed3abd92 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:18.540515 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae3c6679-2c41-43fc-8c8c-af21ed3abd92 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:18.556427 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 363/740] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Sep 25 04:23:18 2022] GET /identity/v3/role_assignments?user.id=e703b42c53974daaa608b0ccc3f018b1&scope.project.id=056ca1b1a9974a7581550de0d9217823&role.id=c51fe7c5ad804040879acfb54269455f => generated 259 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:18.556427 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae3c6679-2c41-43fc-8c8c-af21ed3abd92 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:18.647880 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97e19160-aae1-444a-bd4c-81e191dc7d70 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:18.691042 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2d57f1e-6fb1-473e-a8cb-41242730a103 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:18.691588 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2d57f1e-6fb1-473e-a8cb-41242730a103 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:18.691870 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2d57f1e-6fb1-473e-a8cb-41242730a103 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:18.692196 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2d57f1e-6fb1-473e-a8cb-41242730a103 None admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:18.692976 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2d57f1e-6fb1-473e-a8cb-41242730a103 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:18.693282 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2d57f1e-6fb1-473e-a8cb-41242730a103 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:18.695422 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2d57f1e-6fb1-473e-a8cb-41242730a103 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:18.710957 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 378/741] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Sep 25 04:23:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:18.712269 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66516628-aa40-4752-baae-38f421493b1b None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:18.712558 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66516628-aa40-4752-baae-38f421493b1b None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:18.712857 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66516628-aa40-4752-baae-38f421493b1b None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:18.714556 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 379/742] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:23:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:23:18.717503 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66516628-aa40-4752-baae-38f421493b1b None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:18.722277 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae3c6679-2c41-43fc-8c8c-af21ed3abd92 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:18.731973 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:18.772267 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-688e9f28-6f5f-498a-9354-9e60a45e814b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:18.772952 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-688e9f28-6f5f-498a-9354-9e60a45e814b None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:18.773237 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-688e9f28-6f5f-498a-9354-9e60a45e814b None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:18.773821 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-688e9f28-6f5f-498a-9354-9e60a45e814b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:18.836216 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:18.845969 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66516628-aa40-4752-baae-38f421493b1b None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:18.878688 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-688e9f28-6f5f-498a-9354-9e60a45e814b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:18.880147 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-688e9f28-6f5f-498a-9354-9e60a45e814b None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:18.893699 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-303d11c2-75fd-4710-aee5-e4c51fab07ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:18.894657 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-303d11c2-75fd-4710-aee5-e4c51fab07ec None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:18.895571 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-303d11c2-75fd-4710-aee5-e4c51fab07ec None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:18.896298 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-303d11c2-75fd-4710-aee5-e4c51fab07ec None admin] PATH_INFO: `/v3/projects/service` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:18.905268 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-303d11c2-75fd-4710-aee5-e4c51fab07ec None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:18.907989 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-303d11c2-75fd-4710-aee5-e4c51fab07ec None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:18.924412 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-303d11c2-75fd-4710-aee5-e4c51fab07ec None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 04:23:18.924412 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:18.924412 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:18.924412 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:18.924412 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:18.924412 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:18.924412 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:18.924412 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:18.924412 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:18.924412 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:18.924412 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:18.924412 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:18.924412 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 25 04:23:18.924412 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 25 04:23:18.924412 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 25 04:23:18.925374 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 25 04:23:18.925374 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:18.925374 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:18.925374 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:18.925374 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:18.925374 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:18.925374 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:18.925374 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:18.925374 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:18.925374 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:18.925374 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:18.925374 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:18.925374 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:18.925374 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:18.925374 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:18.925374 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:18.925374 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:18.926288 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 25 04:23:18.926288 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 25 04:23:18.926288 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 25 04:23:18.926288 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 25 04:23:18.926288 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 25 04:23:18.926288 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 25 04:23:18.926288 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 04:23:18.926288 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:23:18.928728 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 380/743] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Sep 25 04:23:18 2022] GET /identity/v3/projects/service => generated 88 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:18.930518 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738d30c2-f8f9-40cd-beb2-85e5064a1cba None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:18.931366 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738d30c2-f8f9-40cd-beb2-85e5064a1cba None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:18.931936 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738d30c2-f8f9-40cd-beb2-85e5064a1cba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:18.956552 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-738d30c2-f8f9-40cd-beb2-85e5064a1cba None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:23:18.958881 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-738d30c2-f8f9-40cd-beb2-85e5064a1cba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:23:18.998109 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 364/744] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Sep 25 04:23:18 2022] GET /identity/v3/auth/tokens => generated 898 bytes in 440 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 25 04:23:18.999159 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-688e9f28-6f5f-498a-9354-9e60a45e814b None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:19.003007 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:19.052662 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 381/745] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:23:18 2022] POST /identity/v3/auth/tokens => generated 898 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 25 04:23:19.052662 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-738d30c2-f8f9-40cd-beb2-85e5064a1cba None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:19.162415 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-688e9f28-6f5f-498a-9354-9e60a45e814b None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:19.213379 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:19.226535 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43d03850-e291-47fa-9040-e42360d37d8e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:19.227156 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d03850-e291-47fa-9040-e42360d37d8e None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:19.227427 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d03850-e291-47fa-9040-e42360d37d8e None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:19.227707 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d03850-e291-47fa-9040-e42360d37d8e None admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:19.228394 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43d03850-e291-47fa-9040-e42360d37d8e None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:19.228723 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43d03850-e291-47fa-9040-e42360d37d8e None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:19.230147 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43d03850-e291-47fa-9040-e42360d37d8e None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:19.246158 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 365/746] 10.222.0.25 () {56 vars in 1148 bytes} [Sun Sep 25 04:23:18 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 244 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:19.248337 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43d03850-e291-47fa-9040-e42360d37d8e None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:19.250986 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-738d30c2-f8f9-40cd-beb2-85e5064a1cba None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:19.315247 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:19.334452 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0149ed53-45a7-4879-8a22-cd98deee3336 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:19.335095 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0149ed53-45a7-4879-8a22-cd98deee3336 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:19.335371 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0149ed53-45a7-4879-8a22-cd98deee3336 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:19.335640 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0149ed53-45a7-4879-8a22-cd98deee3336 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:19.353057 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0149ed53-45a7-4879-8a22-cd98deee3336 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:19.354651 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0149ed53-45a7-4879-8a22-cd98deee3336 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:19.362820 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43d03850-e291-47fa-9040-e42360d37d8e None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:19.372358 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-0149ed53-45a7-4879-8a22-cd98deee3336 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:23:19.372358 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:19.372358 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:19.372358 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:19.372358 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:19.372358 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:19.372358 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:19.372358 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:19.372358 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:19.372358 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:19.372358 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:19.372358 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:19.372358 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 04:23:19.372358 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 04:23:19.372358 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 04:23:19.372358 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 04:23:19.373669 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:19.373669 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:19.373669 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:19.373669 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:19.373669 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:19.373669 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:19.373669 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:19.373669 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:19.373669 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:19.373669 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:19.373669 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:19.373669 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:19.373669 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:19.373669 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:19.373669 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:19.373669 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:19.373669 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 04:23:19.374845 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 04:23:19.374845 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:23:19.374845 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:23:19.375024 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 382/747] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:23:19 2022] GET /identity/v3/domains/Default => generated 87 bytes in 327 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:19.376008 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0149ed53-45a7-4879-8a22-cd98deee3336 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:19.406391 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fcc5a4a-8861-4468-935c-ed62d6fad1ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:19.407126 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fcc5a4a-8861-4468-935c-ed62d6fad1ad None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:19.407466 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fcc5a4a-8861-4468-935c-ed62d6fad1ad None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:19.407810 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fcc5a4a-8861-4468-935c-ed62d6fad1ad None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:19.408594 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fcc5a4a-8861-4468-935c-ed62d6fad1ad None admin] RBAC: Adding query filter params (role.id=c51fe7c5ad804040879acfb54269455f, scope.project.id=056ca1b1a9974a7581550de0d9217823, user.id=23ea88f46e7740d295148a93ade29dd9) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:19.409003 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fcc5a4a-8861-4468-935c-ed62d6fad1ad None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:19.410498 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fcc5a4a-8861-4468-935c-ed62d6fad1ad None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:19.416399 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 366/748] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Sep 25 04:23:19 2022] GET /identity/v3/role_assignments?user.id=23ea88f46e7740d295148a93ade29dd9&scope.project.id=056ca1b1a9974a7581550de0d9217823&role.id=c51fe7c5ad804040879acfb54269455f => generated 259 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:19.417370 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fcc5a4a-8861-4468-935c-ed62d6fad1ad None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:19.509703 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0149ed53-45a7-4879-8a22-cd98deee3336 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:19.584798 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07cb89d5-21a7-4830-b974-107d1b55c731 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:19.585744 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07cb89d5-21a7-4830-b974-107d1b55c731 None admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:19.586208 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07cb89d5-21a7-4830-b974-107d1b55c731 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:19.586665 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07cb89d5-21a7-4830-b974-107d1b55c731 None admin] PATH_INFO: `/v3/services` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:19.587548 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07cb89d5-21a7-4830-b974-107d1b55c731 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:19.589773 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07cb89d5-21a7-4830-b974-107d1b55c731 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:19.621737 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 383/749] 10.222.0.25 () {60 vars in 1141 bytes} [Sun Sep 25 04:23:19 2022] POST /identity/v3/services => generated 251 bytes in 246 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:23:19.625225 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:19.635109 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fcc5a4a-8861-4468-935c-ed62d6fad1ad None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:19.668029 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:19.689452 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5dfe13d-3a3b-4f0c-9df8-7ff84537a8fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:19.690078 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5dfe13d-3a3b-4f0c-9df8-7ff84537a8fb None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:19.690364 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5dfe13d-3a3b-4f0c-9df8-7ff84537a8fb None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:19.690632 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5dfe13d-3a3b-4f0c-9df8-7ff84537a8fb None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:19.691288 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5dfe13d-3a3b-4f0c-9df8-7ff84537a8fb None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:19.691588 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5dfe13d-3a3b-4f0c-9df8-7ff84537a8fb None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:19.693045 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5dfe13d-3a3b-4f0c-9df8-7ff84537a8fb None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:19.711908 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 367/750] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Sep 25 04:23:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 295 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:19.717140 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07cb89d5-21a7-4830-b974-107d1b55c731 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:19.867404 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07cb89d5-21a7-4830-b974-107d1b55c731 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:19.902721 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e780523-e17f-47f6-aceb-db922114489f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:19.903410 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e780523-e17f-47f6-aceb-db922114489f None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:19.903698 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e780523-e17f-47f6-aceb-db922114489f None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:19.903971 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e780523-e17f-47f6-aceb-db922114489f None admin] PATH_INFO: `/v3/projects/service` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:19.910867 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e780523-e17f-47f6-aceb-db922114489f None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:19.912303 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e780523-e17f-47f6-aceb-db922114489f None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:19.923633 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-2e780523-e17f-47f6-aceb-db922114489f None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 04:23:19.923633 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:19.923633 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:19.923633 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:19.923633 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:19.923633 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:19.923633 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:19.923633 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:19.923633 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:19.923633 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:19.923633 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:19.923633 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:19.923633 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 25 04:23:19.923633 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 25 04:23:19.923633 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 25 04:23:19.924545 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 25 04:23:19.924545 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:19.924545 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:19.924545 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:19.924545 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:19.924545 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:19.924545 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:19.924545 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:19.924545 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:19.924545 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:19.924545 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:19.924545 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:19.924545 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:19.924545 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:19.924545 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:19.924545 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:19.924545 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:19.925513 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 25 04:23:19.925513 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 25 04:23:19.925513 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 25 04:23:19.925513 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 25 04:23:19.925513 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 25 04:23:19.925513 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 25 04:23:19.925513 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 04:23:19.925513 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:23:19.925513 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 384/751] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Sep 25 04:23:19 2022] GET /identity/v3/projects/service => generated 88 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:19.930336 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5dfe13d-3a3b-4f0c-9df8-7ff84537a8fb None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:20.054124 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5dfe13d-3a3b-4f0c-9df8-7ff84537a8fb None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:20.096606 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a48d9e0e-62c2-475a-a06c-56ad11e4e1b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:20.097255 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a48d9e0e-62c2-475a-a06c-56ad11e4e1b9 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:20.097535 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a48d9e0e-62c2-475a-a06c-56ad11e4e1b9 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:20.097808 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a48d9e0e-62c2-475a-a06c-56ad11e4e1b9 None admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:20.098501 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a48d9e0e-62c2-475a-a06c-56ad11e4e1b9 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:20.098807 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a48d9e0e-62c2-475a-a06c-56ad11e4e1b9 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:20.100298 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a48d9e0e-62c2-475a-a06c-56ad11e4e1b9 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:20.120188 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 368/752] 10.222.0.25 () {56 vars in 1148 bytes} [Sun Sep 25 04:23:19 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:20.126422 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e780523-e17f-47f6-aceb-db922114489f None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:20.243980 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:20.254122 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d164cb60-cfae-41ab-b891-54bf2b2caa16 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:20.254431 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d164cb60-cfae-41ab-b891-54bf2b2caa16 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:20.254679 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d164cb60-cfae-41ab-b891-54bf2b2caa16 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:20.256271 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 369/753] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:23:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:23:20.262630 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd424a24-ea0a-4655-bad8-fd8e26aca1b7 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:20.268237 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd424a24-ea0a-4655-bad8-fd8e26aca1b7 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:20.268806 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd424a24-ea0a-4655-bad8-fd8e26aca1b7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:20.299375 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-cd424a24-ea0a-4655-bad8-fd8e26aca1b7 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:23:20.302160 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-cd424a24-ea0a-4655-bad8-fd8e26aca1b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:23:20.321799 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e780523-e17f-47f6-aceb-db922114489f None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:20.382177 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f10f9217-689a-4fc6-8b58-39d084fdd4d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:20.383285 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f10f9217-689a-4fc6-8b58-39d084fdd4d9 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:20.385131 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:20.386500 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f10f9217-689a-4fc6-8b58-39d084fdd4d9 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:20.391054 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f10f9217-689a-4fc6-8b58-39d084fdd4d9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:20.392124 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f10f9217-689a-4fc6-8b58-39d084fdd4d9 None admin] RBAC: Adding query filter params (role.id=9b44520f482c4423bd10568d0f86718c, scope.project.id=056ca1b1a9974a7581550de0d9217823, user.id=050b3aab44bb4b17aa4a1ba7cccdb231) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:20.392860 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f10f9217-689a-4fc6-8b58-39d084fdd4d9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:20.394512 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f10f9217-689a-4fc6-8b58-39d084fdd4d9 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:20.404664 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 385/754] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Sep 25 04:23:20 2022] GET /identity/v3/role_assignments?user.id=050b3aab44bb4b17aa4a1ba7cccdb231&scope.project.id=056ca1b1a9974a7581550de0d9217823&role.id=9b44520f482c4423bd10568d0f86718c => generated 259 bytes in 279 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:20.434054 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 370/755] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:23:20 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 25 04:23:20.816033 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f10f9217-689a-4fc6-8b58-39d084fdd4d9 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:20.958792 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f10f9217-689a-4fc6-8b58-39d084fdd4d9 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:20.984919 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:20.992367 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e02ed37-449c-42f6-a84a-d1afe361153b None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:20.992926 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e02ed37-449c-42f6-a84a-d1afe361153b None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:20.993004 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e02ed37-449c-42f6-a84a-d1afe361153b None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:20.995059 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 371/756] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:23:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:23:21.000031 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae68e8d7-cf59-4d89-8a11-6237377d154f None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:21.000262 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae68e8d7-cf59-4d89-8a11-6237377d154f None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:21.000514 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae68e8d7-cf59-4d89-8a11-6237377d154f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:21.002000 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66c1e0c8-f3c3-4160-8857-a5e65343bf26 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:21.002911 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66c1e0c8-f3c3-4160-8857-a5e65343bf26 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:21.003406 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66c1e0c8-f3c3-4160-8857-a5e65343bf26 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:21.003953 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66c1e0c8-f3c3-4160-8857-a5e65343bf26 None admin] PATH_INFO: `/v3/services/network` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:21.004802 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66c1e0c8-f3c3-4160-8857-a5e65343bf26 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:21.006394 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66c1e0c8-f3c3-4160-8857-a5e65343bf26 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:21.025366 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-66c1e0c8-f3c3-4160-8857-a5e65343bf26 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Sep 25 04:23:21.025366 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:21.025366 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 25 04:23:21.025366 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 25 04:23:21.025366 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 25 04:23:21.025366 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 25 04:23:21.025366 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 25 04:23:21.025366 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 04:23:21.025366 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 25 04:23:21.025366 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:23:21.025366 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 25 04:23:21.025366 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:23:21.025366 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:21.025366 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:21.025366 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:21.025366 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:21.025366 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:21.026444 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:21.026444 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:21.026444 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:21.026444 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:21.026444 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:21.026444 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:21.026444 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 25 04:23:21.026444 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 25 04:23:21.026444 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 25 04:23:21.026444 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 25 04:23:21.026444 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:21.026444 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:21.026444 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:21.026444 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:21.026444 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:21.026444 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:21.027309 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:21.027309 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:21.027309 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:21.027309 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:21.027309 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:21.027309 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:21.027309 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:21.027309 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:21.027309 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:21.027309 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:21.027309 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 25 04:23:21.027309 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 04:23:21.027309 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 25 04:23:21.027309 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:23:21.030251 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 386/757] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Sep 25 04:23:20 2022] GET /identity/v3/services/network => generated 88 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:21.040795 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66c1e0c8-f3c3-4160-8857-a5e65343bf26 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:21.045478 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-ae68e8d7-cf59-4d89-8a11-6237377d154f None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:23:21.046933 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-ae68e8d7-cf59-4d89-8a11-6237377d154f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:23:21.189268 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 372/758] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:23:20 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 190 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 04:23:21.214015 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66c1e0c8-f3c3-4160-8857-a5e65343bf26 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:21.248966 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e94c8a20-a5ee-4a32-aa06-0d6b06dd1790 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:21.250500 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e94c8a20-a5ee-4a32-aa06-0d6b06dd1790 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:21.250994 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e94c8a20-a5ee-4a32-aa06-0d6b06dd1790 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:21.251465 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e94c8a20-a5ee-4a32-aa06-0d6b06dd1790 None admin] PATH_INFO: `/v3/services` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:21.252350 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e94c8a20-a5ee-4a32-aa06-0d6b06dd1790 None admin] RBAC: Adding query filter params (name=network) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:21.252879 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e94c8a20-a5ee-4a32-aa06-0d6b06dd1790 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:21.254462 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e94c8a20-a5ee-4a32-aa06-0d6b06dd1790 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:21.259535 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 387/759] 10.222.0.25 () {56 vars in 1112 bytes} [Sun Sep 25 04:23:21 2022] GET /identity/v3/services?name=network => generated 124 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:21.266184 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae68e8d7-cf59-4d89-8a11-6237377d154f None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:21.406979 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae68e8d7-cf59-4d89-8a11-6237377d154f None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:21.410917 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:21.456800 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fff6d61-9c42-4ff9-adbb-0920daeb5164 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:21.457753 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fff6d61-9c42-4ff9-adbb-0920daeb5164 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:21.458239 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fff6d61-9c42-4ff9-adbb-0920daeb5164 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:21.458714 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fff6d61-9c42-4ff9-adbb-0920daeb5164 None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:21.459552 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fff6d61-9c42-4ff9-adbb-0920daeb5164 None admin] RBAC: Adding query filter params (type=network) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:21.460081 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fff6d61-9c42-4ff9-adbb-0920daeb5164 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:21.461747 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fff6d61-9c42-4ff9-adbb-0920daeb5164 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:21.467647 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 373/760] 10.222.0.25 () {56 vars in 1112 bytes} [Sun Sep 25 04:23:21 2022] GET /identity/v3/services?type=network => generated 350 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:21.482015 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e94c8a20-a5ee-4a32-aa06-0d6b06dd1790 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:21.551821 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:21.555651 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fff6d61-9c42-4ff9-adbb-0920daeb5164 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:21.617068 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e94c8a20-a5ee-4a32-aa06-0d6b06dd1790 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:21.680198 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3da641fc-735a-49df-8335-d1108a8d8f43 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:21.680846 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3da641fc-735a-49df-8335-d1108a8d8f43 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:21.681128 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3da641fc-735a-49df-8335-d1108a8d8f43 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:21.681402 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3da641fc-735a-49df-8335-d1108a8d8f43 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:21.682104 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3da641fc-735a-49df-8335-d1108a8d8f43 None admin] RBAC: Adding query filter params (interface=public, service_id=37f55d5557214e3f88d421eb2d3dd608, region_id=RegionOne) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:21.682414 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3da641fc-735a-49df-8335-d1108a8d8f43 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:21.683781 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3da641fc-735a-49df-8335-d1108a8d8f43 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:21.703791 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 388/761] 10.222.0.25 () {56 vars in 1251 bytes} [Sun Sep 25 04:23:21 2022] GET /identity/v3/endpoints?service_id=37f55d5557214e3f88d421eb2d3dd608&interface=public®ion_id=RegionOne => generated 194 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:21.709241 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3da641fc-735a-49df-8335-d1108a8d8f43 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:21.758756 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fff6d61-9c42-4ff9-adbb-0920daeb5164 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:21.817569 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3da641fc-735a-49df-8335-d1108a8d8f43 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:21.833444 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:21.854278 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1030f60-25fc-48ad-b88a-d8b1851952aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:21.855286 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1030f60-25fc-48ad-b88a-d8b1851952aa None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:21.855766 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1030f60-25fc-48ad-b88a-d8b1851952aa None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:21.856237 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1030f60-25fc-48ad-b88a-d8b1851952aa None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:21.856518 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7979744-484a-4571-a750-76418a5b57c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:21.860670 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7979744-484a-4571-a750-76418a5b57c8 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:21.860670 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7979744-484a-4571-a750-76418a5b57c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:21.860670 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7979744-484a-4571-a750-76418a5b57c8 None admin] PATH_INFO: `/v3/services` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:21.860670 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7979744-484a-4571-a750-76418a5b57c8 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:21.860670 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7979744-484a-4571-a750-76418a5b57c8 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:21.866771 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 389/762] 10.222.0.25 () {56 vars in 1087 bytes} [Sun Sep 25 04:23:21 2022] GET /identity/v3/services => generated 772 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:23:21.878361 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1030f60-25fc-48ad-b88a-d8b1851952aa None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 04:23:21.883542 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-a1030f60-25fc-48ad-b88a-d8b1851952aa None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 04:23:21.883542 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:21.883542 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:21.883542 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:21.883542 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:21.883542 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:21.883542 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:21.883542 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:21.883542 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:21.883542 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:21.883542 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:21.883542 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:21.883542 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 04:23:21.883542 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 04:23:21.883542 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 04:23:21.883542 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 04:23:21.884835 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 04:23:21.884835 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 04:23:21.884835 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 04:23:21.884835 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 04:23:21.884835 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:21.884835 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:21.884835 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:21.884835 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:21.884835 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:21.884835 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:21.884835 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:21.884835 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:21.884835 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:21.884835 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:21.884835 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:21.884835 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:21.885926 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:21.885926 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:21.885926 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:21.885926 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:21.885926 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 04:23:21.885926 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 04:23:21.885926 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 04:23:21.885926 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 04:23:21.885926 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 04:23:21.885926 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 04:23:21.885926 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 04:23:21.885926 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:21.887627 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 374/763] 10.222.0.25 () {56 vars in 1096 bytes} [Sun Sep 25 04:23:21 2022] GET /identity/v3/roles/admin => generated 83 bytes in 332 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:21.894455 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7979744-484a-4571-a750-76418a5b57c8 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:21.998194 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a767d8d0-cd21-44b0-ace2-c3046956cddd None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:21.998844 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a767d8d0-cd21-44b0-ace2-c3046956cddd None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:21.998844 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a767d8d0-cd21-44b0-ace2-c3046956cddd None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:22.001117 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 375/764] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:23:21 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:23:22.007780 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43980160-b444-464c-9a8e-54b4c4c5d231 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:22.008449 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43980160-b444-464c-9a8e-54b4c4c5d231 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:22.008890 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43980160-b444-464c-9a8e-54b4c4c5d231 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:22.032961 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-43980160-b444-464c-9a8e-54b4c4c5d231 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:23:22.033342 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7979744-484a-4571-a750-76418a5b57c8 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:22.034748 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-43980160-b444-464c-9a8e-54b4c4c5d231 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:23:22.058936 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:22.111657 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3a34be0-aee7-47e4-aac1-ad992ea9347a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:22.112299 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3a34be0-aee7-47e4-aac1-ad992ea9347a None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:22.112665 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3a34be0-aee7-47e4-aac1-ad992ea9347a None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:22.112985 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3a34be0-aee7-47e4-aac1-ad992ea9347a None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:22.113785 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3a34be0-aee7-47e4-aac1-ad992ea9347a None admin] RBAC: Adding query filter params (name=admin) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:22.114176 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3a34be0-aee7-47e4-aac1-ad992ea9347a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:22.115589 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3a34be0-aee7-47e4-aac1-ad992ea9347a None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:22.116718 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 376/765] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:23:22 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 04:23:22.138793 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 390/766] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Sep 25 04:23:21 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 245 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:22.144891 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43980160-b444-464c-9a8e-54b4c4c5d231 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:22.256054 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43980160-b444-464c-9a8e-54b4c4c5d231 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:22.291490 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e250d9f-bd4f-407f-a37f-ec8cc9983a59 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:22.292457 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e250d9f-bd4f-407f-a37f-ec8cc9983a59 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:22.293039 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e250d9f-bd4f-407f-a37f-ec8cc9983a59 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:22.293490 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e250d9f-bd4f-407f-a37f-ec8cc9983a59 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:22.304831 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e250d9f-bd4f-407f-a37f-ec8cc9983a59 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:22.306532 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e250d9f-bd4f-407f-a37f-ec8cc9983a59 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:22.323661 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-3e250d9f-bd4f-407f-a37f-ec8cc9983a59 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:23:22.323661 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:22.323661 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:22.323661 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:22.323661 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:22.323661 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:22.323661 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:22.323661 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:22.323661 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:22.323661 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:22.323661 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:22.323661 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:22.323661 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 04:23:22.323661 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 04:23:22.323661 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 04:23:22.323661 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 04:23:22.324838 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:22.324838 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:22.324838 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:22.324838 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:22.324838 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:22.324838 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:22.324838 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:22.324838 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:22.324838 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:22.324838 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:22.324838 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:22.324838 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:22.324838 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:22.324838 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:22.324838 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:22.324838 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:22.324838 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 04:23:22.326277 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 04:23:22.326277 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:23:22.326277 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:22.328065 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 377/767] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:23:22 2022] GET /identity/v3/domains/Default => generated 87 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:22.333713 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3a34be0-aee7-47e4-aac1-ad992ea9347a None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:22.379089 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:22.475853 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3a34be0-aee7-47e4-aac1-ad992ea9347a None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:22.508501 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e250d9f-bd4f-407f-a37f-ec8cc9983a59 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:22.529011 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:22.530841 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fde6091-d449-4d1e-9200-88698e311fe2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:22.531664 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fde6091-d449-4d1e-9200-88698e311fe2 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:22.531952 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fde6091-d449-4d1e-9200-88698e311fe2 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:22.532229 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fde6091-d449-4d1e-9200-88698e311fe2 None admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:22.532910 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fde6091-d449-4d1e-9200-88698e311fe2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:22.533218 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fde6091-d449-4d1e-9200-88698e311fe2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:22.534594 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fde6091-d449-4d1e-9200-88698e311fe2 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:22.555941 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 391/768] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Sep 25 04:23:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 223 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:23:22.561568 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fde6091-d449-4d1e-9200-88698e311fe2 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:22.653125 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e250d9f-bd4f-407f-a37f-ec8cc9983a59 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:22.693757 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:22.698175 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fde6091-d449-4d1e-9200-88698e311fe2 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:22.707236 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41b84bd1-cc5b-47d4-9226-3c1203dce304 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:22.708611 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b84bd1-cc5b-47d4-9226-3c1203dce304 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:22.709153 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b84bd1-cc5b-47d4-9226-3c1203dce304 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:22.709621 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b84bd1-cc5b-47d4-9226-3c1203dce304 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:22.721290 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41b84bd1-cc5b-47d4-9226-3c1203dce304 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 04:23:22.726866 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-41b84bd1-cc5b-47d4-9226-3c1203dce304 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 04:23:22.726866 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:22.726866 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:22.726866 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:22.726866 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:22.726866 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:22.726866 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:22.726866 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:22.726866 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:22.726866 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:22.726866 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:22.726866 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:22.726866 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 04:23:22.726866 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 04:23:22.726866 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 04:23:22.726866 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 04:23:22.727827 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 04:23:22.727827 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 04:23:22.727827 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 04:23:22.727827 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 04:23:22.727827 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:22.727827 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:22.727827 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:22.727827 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:22.727827 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:22.727827 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:22.727827 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:22.727827 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:22.727827 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:22.727827 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:22.727827 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:22.727827 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:22.728782 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:22.728782 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:22.728782 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:22.728782 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:22.728782 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 04:23:22.728782 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 04:23:22.728782 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 04:23:22.728782 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 04:23:22.728782 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 04:23:22.728782 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 04:23:22.728782 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 04:23:22.728782 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:22.731499 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 378/769] 10.222.0.25 () {56 vars in 1096 bytes} [Sun Sep 25 04:23:22 2022] GET /identity/v3/roles/admin => generated 83 bytes in 223 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:22.733069 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0bf80c0-842c-43a0-80c2-2be4bc491e5d None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:22.733604 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0bf80c0-842c-43a0-80c2-2be4bc491e5d None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:22.734821 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0bf80c0-842c-43a0-80c2-2be4bc491e5d None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:22.740453 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 379/770] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:23:22 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:23:22.741427 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0bf80c0-842c-43a0-80c2-2be4bc491e5d None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:22.742526 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64063f00-a05a-4089-8d8e-3ee151b87099 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:22.743287 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64063f00-a05a-4089-8d8e-3ee151b87099 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:22.743718 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64063f00-a05a-4089-8d8e-3ee151b87099 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:22.744133 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64063f00-a05a-4089-8d8e-3ee151b87099 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:22.753987 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64063f00-a05a-4089-8d8e-3ee151b87099 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:22.755635 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64063f00-a05a-4089-8d8e-3ee151b87099 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:22.767439 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-64063f00-a05a-4089-8d8e-3ee151b87099 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 25 04:23:22.767439 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:22.767439 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:22.767439 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:22.767439 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:22.767439 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:22.767439 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:22.767439 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:22.767439 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:22.767439 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:22.767439 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:22.767439 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:22.767439 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 25 04:23:22.767439 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 25 04:23:22.767439 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 25 04:23:22.767439 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 25 04:23:22.768596 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:22.768596 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:22.768596 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 25 04:23:22.768596 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 04:23:22.768596 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 25 04:23:22.768596 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 04:23:22.768596 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:22.768596 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:22.768596 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:22.768596 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:22.768596 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:22.768596 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:22.768596 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:22.768596 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:22.768596 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:22.768596 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:22.768596 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:22.768596 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:22.769628 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:22.769628 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:22.769628 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 25 04:23:22.769628 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 25 04:23:22.769628 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 25 04:23:22.769628 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 25 04:23:22.769628 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 25 04:23:22.769628 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 25 04:23:22.769628 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 25 04:23:22.769628 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:23:22.774481 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 392/771] 10.222.0.25 () {56 vars in 1093 bytes} [Sun Sep 25 04:23:22 2022] GET /identity/v3/users/nova => generated 82 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 04:23:22.775765 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45c0d329-b522-4192-8812-bcc39d6ff9ae None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:22.776123 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45c0d329-b522-4192-8812-bcc39d6ff9ae None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:22.776586 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45c0d329-b522-4192-8812-bcc39d6ff9ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:22.796416 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-45c0d329-b522-4192-8812-bcc39d6ff9ae None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:23:22.797758 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-45c0d329-b522-4192-8812-bcc39d6ff9ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:23:22.826217 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:22.872509 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0bf80c0-842c-43a0-80c2-2be4bc491e5d None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:22.875037 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 393/772] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:23:22 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 04:23:22.876707 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45c0d329-b522-4192-8812-bcc39d6ff9ae None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:22.956158 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a89196ec-4ff3-4ffb-a574-d4ad663bda8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:22.957132 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a89196ec-4ff3-4ffb-a574-d4ad663bda8d None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:22.957271 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a89196ec-4ff3-4ffb-a574-d4ad663bda8d None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:22.957505 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a89196ec-4ff3-4ffb-a574-d4ad663bda8d None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:22.958246 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a89196ec-4ff3-4ffb-a574-d4ad663bda8d None admin] RBAC: Adding query filter params (name=admin) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:22.958623 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a89196ec-4ff3-4ffb-a574-d4ad663bda8d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:22.960075 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a89196ec-4ff3-4ffb-a574-d4ad663bda8d None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:22.980501 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 380/773] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Sep 25 04:23:22 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 239 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:22.986062 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:22.987159 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a89196ec-4ff3-4ffb-a574-d4ad663bda8d None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:22.996783 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45c0d329-b522-4192-8812-bcc39d6ff9ae None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:23.038067 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03faf028-ba5b-4b52-8635-a2e7f82881e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:23.038966 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03faf028-ba5b-4b52-8635-a2e7f82881e4 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:23.039440 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03faf028-ba5b-4b52-8635-a2e7f82881e4 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:23.039882 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03faf028-ba5b-4b52-8635-a2e7f82881e4 None admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:23.040827 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03faf028-ba5b-4b52-8635-a2e7f82881e4 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:23.041338 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03faf028-ba5b-4b52-8635-a2e7f82881e4 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:23.043457 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03faf028-ba5b-4b52-8635-a2e7f82881e4 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:23.064084 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 394/774] 10.222.0.25 () {56 vars in 1133 bytes} [Sun Sep 25 04:23:22 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:23.071536 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03faf028-ba5b-4b52-8635-a2e7f82881e4 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:23.214445 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a89196ec-4ff3-4ffb-a574-d4ad663bda8d None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:23.245280 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03faf028-ba5b-4b52-8635-a2e7f82881e4 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:23.246019 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:23.253975 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4135ca61-ba95-40fe-bbe8-7cc75de2cd9a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:23.254894 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4135ca61-ba95-40fe-bbe8-7cc75de2cd9a None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:23.255372 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4135ca61-ba95-40fe-bbe8-7cc75de2cd9a None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:23.255858 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4135ca61-ba95-40fe-bbe8-7cc75de2cd9a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:23.267198 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4135ca61-ba95-40fe-bbe8-7cc75de2cd9a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:23.268952 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4135ca61-ba95-40fe-bbe8-7cc75de2cd9a None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:23.283638 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-4135ca61-ba95-40fe-bbe8-7cc75de2cd9a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:23:23.283638 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:23.283638 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:23.283638 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:23.283638 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:23.283638 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:23.283638 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:23.283638 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:23.283638 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:23.283638 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:23.283638 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:23.283638 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:23.283638 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 04:23:23.283638 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 04:23:23.283638 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 04:23:23.283638 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 04:23:23.284587 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:23.284587 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:23.284587 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:23.284587 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:23.284587 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:23.284587 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:23.284587 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:23.284587 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:23.284587 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:23.284587 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:23.284587 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:23.284587 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:23.284587 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:23.284587 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:23.284587 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:23.284587 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:23.284587 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 04:23:23.285534 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 04:23:23.285534 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:23:23.285534 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:23.285707 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-703226de-3b8d-4a12-a676-053652874515 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:23.285707 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-703226de-3b8d-4a12-a676-053652874515 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:23.285707 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-703226de-3b8d-4a12-a676-053652874515 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:23.285707 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-703226de-3b8d-4a12-a676-053652874515 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:23.286424 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 381/775] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:23:22 2022] GET /identity/v3/domains/Default => generated 87 bytes in 299 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:23.287443 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29d92a2f-394e-418e-8b93-0e27f7e41b81 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:23.287750 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29d92a2f-394e-418e-8b93-0e27f7e41b81 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:23.288025 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29d92a2f-394e-418e-8b93-0e27f7e41b81 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:23.292813 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 382/776] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:23:23 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:23:23.293557 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29d92a2f-394e-418e-8b93-0e27f7e41b81 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:23.303579 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-703226de-3b8d-4a12-a676-053652874515 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:23.305074 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-703226de-3b8d-4a12-a676-053652874515 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:23.408117 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29d92a2f-394e-418e-8b93-0e27f7e41b81 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:23.428918 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:23.435446 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-703226de-3b8d-4a12-a676-053652874515 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:23:23.435446 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:23.435446 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:23.435446 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:23.435446 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:23.435446 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:23.435446 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:23.435446 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:23.435446 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:23.435446 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:23.435446 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:23.435446 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:23.435446 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 04:23:23.435446 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 04:23:23.435446 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 04:23:23.435446 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 04:23:23.440784 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:23.440784 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:23.440784 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:23.440784 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:23.440784 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:23.440784 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:23.440784 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:23.440784 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:23.440784 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:23.440784 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:23.440784 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:23.440784 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:23.440784 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:23.440784 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:23.440784 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:23.440784 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:23.440784 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 04:23:23.441740 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 04:23:23.441740 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:23:23.441740 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:23:23.441740 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 395/777] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:23:23 2022] GET /identity/v3/domains/Default => generated 87 bytes in 367 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:23.441740 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-703226de-3b8d-4a12-a676-053652874515 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:23.454087 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a5b9843-c2b7-4a72-af53-2ef3137eee96 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:23.454732 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a5b9843-c2b7-4a72-af53-2ef3137eee96 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:23.455041 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a5b9843-c2b7-4a72-af53-2ef3137eee96 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:23.455314 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a5b9843-c2b7-4a72-af53-2ef3137eee96 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:23.455893 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a5b9843-c2b7-4a72-af53-2ef3137eee96 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:23.457339 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a5b9843-c2b7-4a72-af53-2ef3137eee96 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:23.470467 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-6a5b9843-c2b7-4a72-af53-2ef3137eee96 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 25 04:23:23.470467 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:23.470467 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 25 04:23:23.470467 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 25 04:23:23.470467 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 25 04:23:23.470467 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 25 04:23:23.470467 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 25 04:23:23.470467 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 04:23:23.470467 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 25 04:23:23.470467 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:23.470467 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 25 04:23:23.470467 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:23.470467 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:23.470467 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:23.470467 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:23.470467 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:23.470467 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:23.471518 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:23.471518 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:23.471518 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:23.471518 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:23.471518 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:23.471518 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:23.471518 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 25 04:23:23.471518 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 25 04:23:23.471518 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 25 04:23:23.471518 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 25 04:23:23.471518 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:23.471518 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:23.471518 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:23.471518 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:23.471518 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:23.471518 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:23.472375 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:23.472375 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:23.472375 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:23.472375 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:23.472375 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:23.472375 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:23.472375 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:23.472375 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:23.472375 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:23.472375 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:23.472375 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 25 04:23:23.472375 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 04:23:23.472375 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 25 04:23:23.472375 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:23.472375 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 383/778] 10.222.0.25 () {56 vars in 1129 bytes} [Sun Sep 25 04:23:23 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:23.473306 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18c3b085-ea06-4094-89cb-3dd408330884 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:23.473306 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18c3b085-ea06-4094-89cb-3dd408330884 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:23.473306 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18c3b085-ea06-4094-89cb-3dd408330884 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:23.495573 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-18c3b085-ea06-4094-89cb-3dd408330884 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:23:23.496793 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-18c3b085-ea06-4094-89cb-3dd408330884 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:23:23.542589 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:23.564046 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-703226de-3b8d-4a12-a676-053652874515 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:23.580660 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 384/779] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:23:23 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 04:23:23.580660 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18c3b085-ea06-4094-89cb-3dd408330884 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:23.615696 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1a66405-c7c3-4ae7-92ad-9638c2ea72e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:23.616311 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1a66405-c7c3-4ae7-92ad-9638c2ea72e2 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:23.616932 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1a66405-c7c3-4ae7-92ad-9638c2ea72e2 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:23.617474 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1a66405-c7c3-4ae7-92ad-9638c2ea72e2 None admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:23.618125 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1a66405-c7c3-4ae7-92ad-9638c2ea72e2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:23.618430 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1a66405-c7c3-4ae7-92ad-9638c2ea72e2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:23.620038 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1a66405-c7c3-4ae7-92ad-9638c2ea72e2 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:23.640805 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 396/780] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Sep 25 04:23:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:23.641808 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1a66405-c7c3-4ae7-92ad-9638c2ea72e2 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:23.710553 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18c3b085-ea06-4094-89cb-3dd408330884 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:23.722343 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:23.748061 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff0928b3-7652-4a77-b013-7376ba2cff7b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:23.751313 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff0928b3-7652-4a77-b013-7376ba2cff7b None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:23.752086 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff0928b3-7652-4a77-b013-7376ba2cff7b None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:23.752567 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff0928b3-7652-4a77-b013-7376ba2cff7b None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:23.753618 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff0928b3-7652-4a77-b013-7376ba2cff7b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:23.754416 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff0928b3-7652-4a77-b013-7376ba2cff7b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:23.755996 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff0928b3-7652-4a77-b013-7376ba2cff7b None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:23.769818 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1a66405-c7c3-4ae7-92ad-9638c2ea72e2 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:23.771925 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 385/781] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Sep 25 04:23:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:23.773540 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff0928b3-7652-4a77-b013-7376ba2cff7b None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:23.807852 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79846de7-6b5f-4061-9a8f-a50a9f2594c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:23.808459 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79846de7-6b5f-4061-9a8f-a50a9f2594c6 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:23.808804 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79846de7-6b5f-4061-9a8f-a50a9f2594c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:23.809077 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79846de7-6b5f-4061-9a8f-a50a9f2594c6 None admin] PATH_INFO: `/v3/services` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:23.809816 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79846de7-6b5f-4061-9a8f-a50a9f2594c6 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:23.810098 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79846de7-6b5f-4061-9a8f-a50a9f2594c6 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:23.811457 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79846de7-6b5f-4061-9a8f-a50a9f2594c6 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:23.821035 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 397/782] 10.222.0.25 () {56 vars in 1124 bytes} [Sun Sep 25 04:23:23 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:23.821035 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79846de7-6b5f-4061-9a8f-a50a9f2594c6 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:23.822911 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:23.901583 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff0928b3-7652-4a77-b013-7376ba2cff7b None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:23.933013 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79846de7-6b5f-4061-9a8f-a50a9f2594c6 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:23.939171 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-008b8bb3-bdc6-4567-87db-18fdf724cee8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:23.940465 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-008b8bb3-bdc6-4567-87db-18fdf724cee8 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:23.940947 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-008b8bb3-bdc6-4567-87db-18fdf724cee8 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:23.941393 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-008b8bb3-bdc6-4567-87db-18fdf724cee8 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:23.951719 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-008b8bb3-bdc6-4567-87db-18fdf724cee8 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:23.954782 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-008b8bb3-bdc6-4567-87db-18fdf724cee8 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:23.956475 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:23.967746 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-172bafde-ab06-473b-aecd-6984268cf1f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:23.968566 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-172bafde-ab06-473b-aecd-6984268cf1f9 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:23.968860 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-172bafde-ab06-473b-aecd-6984268cf1f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:23.969132 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-172bafde-ab06-473b-aecd-6984268cf1f9 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:23.969503 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-008b8bb3-bdc6-4567-87db-18fdf724cee8 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 25 04:23:23.969503 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:23.969503 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:23.969503 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:23.969503 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:23.969503 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:23.969503 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:23.969503 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:23.969503 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:23.969503 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:23.969503 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:23.969503 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:23.969503 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 25 04:23:23.969503 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 25 04:23:23.969503 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 25 04:23:23.969503 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 25 04:23:23.970413 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:23.970413 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:23.970413 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 25 04:23:23.970413 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 04:23:23.970413 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 25 04:23:23.970413 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 04:23:23.970413 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:23.970413 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:23.970413 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:23.970413 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:23.970413 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:23.970413 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:23.970413 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:23.970413 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:23.970413 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:23.970413 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:23.970413 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:23.970413 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:23.976372 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:23.976372 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:23.976372 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 25 04:23:23.976372 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 25 04:23:23.976372 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 25 04:23:23.976372 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 25 04:23:23.976372 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 25 04:23:23.976372 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 25 04:23:23.976372 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 25 04:23:23.976372 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:23.976372 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 386/783] 10.222.0.25 () {56 vars in 1096 bytes} [Sun Sep 25 04:23:23 2022] GET /identity/v3/users/swift => generated 83 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:23.976372 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-008b8bb3-bdc6-4567-87db-18fdf724cee8 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:23.977099 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-172bafde-ab06-473b-aecd-6984268cf1f9 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:23.977624 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-172bafde-ab06-473b-aecd-6984268cf1f9 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:23.989768 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-172bafde-ab06-473b-aecd-6984268cf1f9 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 04:23:23.989768 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:23.989768 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:23.989768 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:23.989768 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:23.989768 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:23.989768 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:23.989768 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:23.989768 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:23.989768 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:23.989768 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:23.989768 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:23.989768 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 25 04:23:23.989768 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 25 04:23:23.989768 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 25 04:23:23.990673 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 25 04:23:23.990673 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:23.990673 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:23.990673 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:23.990673 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:23.990673 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:23.990673 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:23.990673 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:23.990673 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:23.990673 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:23.990673 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:23.990673 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:23.990673 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:23.990673 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:23.990673 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:23.990673 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:23.990673 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:23.991594 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 25 04:23:23.991594 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 25 04:23:23.991594 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 25 04:23:23.991594 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 25 04:23:23.991594 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 25 04:23:23.991594 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 25 04:23:23.991594 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 04:23:23.991594 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:23:23.991594 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 398/784] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Sep 25 04:23:23 2022] GET /identity/v3/projects/service => generated 88 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:23.992118 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-172bafde-ab06-473b-aecd-6984268cf1f9 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:24.091737 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:24.096403 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-172bafde-ab06-473b-aecd-6984268cf1f9 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:24.099698 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-008b8bb3-bdc6-4567-87db-18fdf724cee8 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:24.132466 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90a56e6c-0c26-43b0-8b45-08d464e15533 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:24.133135 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a56e6c-0c26-43b0-8b45-08d464e15533 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:24.133407 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a56e6c-0c26-43b0-8b45-08d464e15533 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:24.133673 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a56e6c-0c26-43b0-8b45-08d464e15533 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:24.137124 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8128b976-bb48-4850-8b7f-c74a99cb49fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:24.137737 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8128b976-bb48-4850-8b7f-c74a99cb49fa None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:24.138016 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8128b976-bb48-4850-8b7f-c74a99cb49fa None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:24.138285 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8128b976-bb48-4850-8b7f-c74a99cb49fa None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:24.138904 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8128b976-bb48-4850-8b7f-c74a99cb49fa None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:24.139205 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8128b976-bb48-4850-8b7f-c74a99cb49fa None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:24.141560 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8128b976-bb48-4850-8b7f-c74a99cb49fa None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:24.144845 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90a56e6c-0c26-43b0-8b45-08d464e15533 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 04:23:24.146577 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 387/785] 10.222.0.25 () {56 vars in 1124 bytes} [Sun Sep 25 04:23:23 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:24.149513 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8128b976-bb48-4850-8b7f-c74a99cb49fa None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:24.150020 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-90a56e6c-0c26-43b0-8b45-08d464e15533 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 25 04:23:24.150020 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:24.150020 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:24.150020 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:24.150020 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:24.150020 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:24.150020 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:24.150020 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:24.150020 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:24.150020 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:24.150020 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:24.150020 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:24.150020 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 04:23:24.150020 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 04:23:24.150020 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 04:23:24.150020 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 04:23:24.150950 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 04:23:24.150950 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 04:23:24.150950 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 04:23:24.150950 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 04:23:24.150950 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:24.150950 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:24.150950 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:24.150950 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:24.150950 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:24.150950 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:24.150950 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:24.150950 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:24.150950 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:24.150950 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:24.150950 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:24.150950 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:24.151857 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:24.151857 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:24.151857 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:24.151857 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:24.151857 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 04:23:24.151857 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 04:23:24.151857 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 04:23:24.151857 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 04:23:24.151857 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 04:23:24.151857 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 04:23:24.151857 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 25 04:23:24.151857 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:23:24.156121 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 399/786] 10.222.0.25 () {56 vars in 1102 bytes} [Sun Sep 25 04:23:23 2022] GET /identity/v3/roles/service => generated 85 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:24.157061 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90a56e6c-0c26-43b0-8b45-08d464e15533 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:24.205978 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:24.258548 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8128b976-bb48-4850-8b7f-c74a99cb49fa None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:24.268351 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90a56e6c-0c26-43b0-8b45-08d464e15533 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:24.296784 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-254eedc6-bfd0-44f3-8e77-f9c87c8b5b9a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:24.297429 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-254eedc6-bfd0-44f3-8e77-f9c87c8b5b9a None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:24.297715 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-254eedc6-bfd0-44f3-8e77-f9c87c8b5b9a None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:24.297983 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-254eedc6-bfd0-44f3-8e77-f9c87c8b5b9a None admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:24.298739 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-254eedc6-bfd0-44f3-8e77-f9c87c8b5b9a None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:24.299050 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-254eedc6-bfd0-44f3-8e77-f9c87c8b5b9a None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:24.300544 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-254eedc6-bfd0-44f3-8e77-f9c87c8b5b9a None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:24.306359 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92d460ec-1706-46d6-94ff-f896d627e640 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:24.306986 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92d460ec-1706-46d6-94ff-f896d627e640 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:24.307259 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92d460ec-1706-46d6-94ff-f896d627e640 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:24.307528 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92d460ec-1706-46d6-94ff-f896d627e640 None admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:24.308226 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92d460ec-1706-46d6-94ff-f896d627e640 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:24.308529 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92d460ec-1706-46d6-94ff-f896d627e640 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:24.310027 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92d460ec-1706-46d6-94ff-f896d627e640 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:24.321529 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 388/787] 10.222.0.25 () {56 vars in 1135 bytes} [Sun Sep 25 04:23:24 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:24.322378 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-254eedc6-bfd0-44f3-8e77-f9c87c8b5b9a None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:24.329397 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 400/788] 10.222.0.25 () {56 vars in 1148 bytes} [Sun Sep 25 04:23:24 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:24.330535 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92d460ec-1706-46d6-94ff-f896d627e640 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:24.341408 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:24.432227 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-254eedc6-bfd0-44f3-8e77-f9c87c8b5b9a None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:24.443564 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92d460ec-1706-46d6-94ff-f896d627e640 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:24.446168 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:24.465704 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31a1ae5b-d2ec-47de-b5c3-516889f5055b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:24.466317 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a1ae5b-d2ec-47de-b5c3-516889f5055b None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:24.466606 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a1ae5b-d2ec-47de-b5c3-516889f5055b None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:24.466874 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a1ae5b-d2ec-47de-b5c3-516889f5055b None admin] PATH_INFO: `/v3/endpoints` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:24.467572 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31a1ae5b-d2ec-47de-b5c3-516889f5055b None admin] RBAC: Adding query filter params (interface=public, service_id=e2af0eafe2564763b726c22314818eca, region_id=RegionOne) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:24.468401 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31a1ae5b-d2ec-47de-b5c3-516889f5055b None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:24.469827 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31a1ae5b-d2ec-47de-b5c3-516889f5055b None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:24.477229 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 389/789] 10.222.0.25 () {56 vars in 1251 bytes} [Sun Sep 25 04:23:24 2022] GET /identity/v3/endpoints?service_id=e2af0eafe2564763b726c22314818eca&interface=public®ion_id=RegionOne => generated 194 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:24.478020 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31a1ae5b-d2ec-47de-b5c3-516889f5055b None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:24.481908 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f689381f-c154-4b1f-a020-5860ca6c8d60 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:24.482532 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f689381f-c154-4b1f-a020-5860ca6c8d60 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:24.482807 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f689381f-c154-4b1f-a020-5860ca6c8d60 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:24.483072 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f689381f-c154-4b1f-a020-5860ca6c8d60 None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:24.484342 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f689381f-c154-4b1f-a020-5860ca6c8d60 None admin] RBAC: Adding query filter params (name=service) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:24.484662 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f689381f-c154-4b1f-a020-5860ca6c8d60 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:24.486159 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f689381f-c154-4b1f-a020-5860ca6c8d60 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:24.494439 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 401/790] 10.222.0.25 () {56 vars in 1103 bytes} [Sun Sep 25 04:23:24 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:24.495049 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f689381f-c154-4b1f-a020-5860ca6c8d60 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:24.563322 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:24.581867 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31a1ae5b-d2ec-47de-b5c3-516889f5055b None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:24.592540 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f689381f-c154-4b1f-a020-5860ca6c8d60 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:24.615208 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f7897cd-596d-400c-a1ec-86e4068cd569 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:24.615834 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f7897cd-596d-400c-a1ec-86e4068cd569 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:24.616119 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f7897cd-596d-400c-a1ec-86e4068cd569 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:24.616393 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f7897cd-596d-400c-a1ec-86e4068cd569 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:24.625799 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fc33a3d-dc64-4d25-a9c8-cfa0e582217f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:24.626458 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fc33a3d-dc64-4d25-a9c8-cfa0e582217f None admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:24.626733 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fc33a3d-dc64-4d25-a9c8-cfa0e582217f None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:24.627002 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fc33a3d-dc64-4d25-a9c8-cfa0e582217f None admin] PATH_INFO: `/v3/projects/056ca1b1a9974a7581550de0d9217823/users/e703b42c53974daaa608b0ccc3f018b1/roles/c51fe7c5ad804040879acfb54269455f` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:24.627403 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f7897cd-596d-400c-a1ec-86e4068cd569 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:24.628880 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f7897cd-596d-400c-a1ec-86e4068cd569 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:24.629003 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fc33a3d-dc64-4d25-a9c8-cfa0e582217f None admin] RBAC: Authorizing `identity:create_grant(project_id=056ca1b1a9974a7581550de0d9217823, user_id=e703b42c53974daaa608b0ccc3f018b1, role_id=c51fe7c5ad804040879acfb54269455f)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:24.630319 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fc33a3d-dc64-4d25-a9c8-cfa0e582217f None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:24.643225 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 402/791] 10.222.0.25 () {58 vars in 1439 bytes} [Sun Sep 25 04:23:24 2022] PUT /identity/v3/projects/056ca1b1a9974a7581550de0d9217823/users/e703b42c53974daaa608b0ccc3f018b1/roles/c51fe7c5ad804040879acfb54269455f => generated 0 bytes in 148 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:23:24.643875 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-3f7897cd-596d-400c-a1ec-86e4068cd569 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:23:24.643875 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:24.643875 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:24.643875 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:24.643875 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:24.643875 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:24.643875 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:24.643875 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:24.643875 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:24.643875 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:24.643875 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:24.643875 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:24.643875 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 04:23:24.643875 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 04:23:24.643875 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 04:23:24.643875 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 04:23:24.644861 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:24.644861 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:24.644861 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:24.644861 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:24.644861 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:24.644861 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:24.644861 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:24.644861 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:24.644861 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:24.644861 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:24.644861 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:24.644861 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:24.644861 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:24.644861 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:24.644861 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:24.644861 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:24.644861 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 04:23:24.645806 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 04:23:24.645806 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:23:24.645806 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:24.645996 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf97101-624d-4ca4-8b76-1a2ea4235c8d None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:24.645996 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf97101-624d-4ca4-8b76-1a2ea4235c8d None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:24.645996 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf97101-624d-4ca4-8b76-1a2ea4235c8d None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:24.646181 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 390/792] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:23:24 2022] GET /identity/v3/domains/Default => generated 87 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:24.646181 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f7897cd-596d-400c-a1ec-86e4068cd569 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:24.646795 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 403/793] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:23:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:23:24.647213 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbf97101-624d-4ca4-8b76-1a2ea4235c8d None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:24.680789 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:24.769836 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f7897cd-596d-400c-a1ec-86e4068cd569 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:24.770758 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbf97101-624d-4ca4-8b76-1a2ea4235c8d None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:24.808626 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65523181-2d53-4c4d-9e60-6cd520385abd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:24.809279 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65523181-2d53-4c4d-9e60-6cd520385abd None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:24.809564 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65523181-2d53-4c4d-9e60-6cd520385abd None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:24.809842 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65523181-2d53-4c4d-9e60-6cd520385abd None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:24.810029 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a098c32-ef83-4707-a02a-9a76f2a6654e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:24.810461 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65523181-2d53-4c4d-9e60-6cd520385abd None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:24.810918 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a098c32-ef83-4707-a02a-9a76f2a6654e None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:24.811291 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a098c32-ef83-4707-a02a-9a76f2a6654e None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:24.811744 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a098c32-ef83-4707-a02a-9a76f2a6654e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:24.811923 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65523181-2d53-4c4d-9e60-6cd520385abd None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:24.813234 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:24.824307 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 391/794] 10.222.0.25 () {56 vars in 1087 bytes} [Sun Sep 25 04:23:24 2022] GET /identity/v3/services => generated 772 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:24.824307 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52d6413b-3dd1-495c-a9cf-06f4cf2d1b2d None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:24.824307 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52d6413b-3dd1-495c-a9cf-06f4cf2d1b2d None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:24.824307 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52d6413b-3dd1-495c-a9cf-06f4cf2d1b2d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:24.832196 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a098c32-ef83-4707-a02a-9a76f2a6654e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:24.833769 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a098c32-ef83-4707-a02a-9a76f2a6654e None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:24.851398 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-52d6413b-3dd1-495c-a9cf-06f4cf2d1b2d None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:23:24.852625 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-52d6413b-3dd1-495c-a9cf-06f4cf2d1b2d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:23:24.854764 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-9a098c32-ef83-4707-a02a-9a76f2a6654e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:23:24.854764 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:24.854764 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:24.854764 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:24.854764 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:24.854764 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:24.854764 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:24.854764 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:24.854764 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:24.854764 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:24.854764 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:24.854764 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:24.854764 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 04:23:24.854764 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 04:23:24.854764 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 04:23:24.854764 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 04:23:24.855953 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:24.855953 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:24.855953 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:24.855953 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:24.855953 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:24.855953 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:24.855953 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:24.855953 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:24.855953 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:24.855953 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:24.855953 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:24.855953 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:24.855953 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:24.855953 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:24.855953 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:24.855953 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:24.855953 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 04:23:24.857161 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 04:23:24.857161 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:23:24.857161 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:23:24.860733 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 404/795] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:23:24 2022] GET /identity/v3/domains/Default => generated 87 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:24.861950 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a098c32-ef83-4707-a02a-9a76f2a6654e None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:24.946570 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 392/796] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:23:24 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 04:23:24.947606 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52d6413b-3dd1-495c-a9cf-06f4cf2d1b2d None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:24.966361 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:24.992386 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a098c32-ef83-4707-a02a-9a76f2a6654e None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:25.031784 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6127d99-18d3-4c4b-9aea-de5bb334304a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:25.032328 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6127d99-18d3-4c4b-9aea-de5bb334304a None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:25.032608 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6127d99-18d3-4c4b-9aea-de5bb334304a None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:25.032910 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6127d99-18d3-4c4b-9aea-de5bb334304a None admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:25.033557 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6127d99-18d3-4c4b-9aea-de5bb334304a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:25.033858 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6127d99-18d3-4c4b-9aea-de5bb334304a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:25.035230 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6127d99-18d3-4c4b-9aea-de5bb334304a None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:25.048774 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 405/797] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Sep 25 04:23:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:25.054600 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6127d99-18d3-4c4b-9aea-de5bb334304a None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:25.103999 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52d6413b-3dd1-495c-a9cf-06f4cf2d1b2d None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:25.140137 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:25.151528 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a8c03df-91df-4443-b0e5-5150316949ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:25.152741 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a8c03df-91df-4443-b0e5-5150316949ae None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:25.153200 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a8c03df-91df-4443-b0e5-5150316949ae None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:25.153622 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a8c03df-91df-4443-b0e5-5150316949ae None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:25.154414 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a8c03df-91df-4443-b0e5-5150316949ae None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:25.155933 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a8c03df-91df-4443-b0e5-5150316949ae None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:25.157776 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a8c03df-91df-4443-b0e5-5150316949ae None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:25.168576 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6127d99-18d3-4c4b-9aea-de5bb334304a None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:25.174283 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 393/798] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Sep 25 04:23:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:25.179441 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a8c03df-91df-4443-b0e5-5150316949ae None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:25.231888 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d73ffb03-6aae-42d1-8718-e179eb89d500 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:25.232679 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d73ffb03-6aae-42d1-8718-e179eb89d500 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:25.232837 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d73ffb03-6aae-42d1-8718-e179eb89d500 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:25.233088 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d73ffb03-6aae-42d1-8718-e179eb89d500 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:25.243562 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d73ffb03-6aae-42d1-8718-e179eb89d500 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:25.245149 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d73ffb03-6aae-42d1-8718-e179eb89d500 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:25.254362 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:25.271181 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-d73ffb03-6aae-42d1-8718-e179eb89d500 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 04:23:25.271181 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:25.271181 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:25.271181 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:25.271181 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:25.271181 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:25.271181 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:25.271181 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:25.271181 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:25.271181 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:25.271181 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:25.271181 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:25.271181 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 25 04:23:25.271181 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 25 04:23:25.271181 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 25 04:23:25.272369 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 25 04:23:25.272369 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:25.272369 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:25.272369 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:25.272369 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:25.272369 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:25.272369 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:25.272369 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:25.272369 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:25.272369 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:25.272369 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:25.272369 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:25.272369 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:25.272369 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:25.272369 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:25.272369 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:25.272369 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:25.273692 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 25 04:23:25.273692 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 25 04:23:25.273692 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 25 04:23:25.273692 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 25 04:23:25.273692 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 25 04:23:25.273692 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 25 04:23:25.273692 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 04:23:25.273692 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:23:25.273692 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 406/799] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Sep 25 04:23:25 2022] GET /identity/v3/projects/service => generated 88 bytes in 218 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 04:23:25.278019 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d73ffb03-6aae-42d1-8718-e179eb89d500 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:25.314823 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a8c03df-91df-4443-b0e5-5150316949ae None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:25.362119 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28ae5dfa-cc45-4e0c-b3f0-899082f1275d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:25.362786 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28ae5dfa-cc45-4e0c-b3f0-899082f1275d None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:25.363061 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28ae5dfa-cc45-4e0c-b3f0-899082f1275d None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:25.363331 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28ae5dfa-cc45-4e0c-b3f0-899082f1275d None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:25.373588 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28ae5dfa-cc45-4e0c-b3f0-899082f1275d None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:25.375240 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28ae5dfa-cc45-4e0c-b3f0-899082f1275d None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:25.393950 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-28ae5dfa-cc45-4e0c-b3f0-899082f1275d None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 25 04:23:25.393950 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:25.393950 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:25.393950 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:25.393950 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:25.393950 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:25.393950 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:25.393950 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:25.393950 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:25.393950 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:25.393950 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:25.393950 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:25.393950 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 25 04:23:25.393950 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 25 04:23:25.393950 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 25 04:23:25.393950 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 25 04:23:25.394969 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:25.394969 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:25.394969 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 25 04:23:25.394969 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 04:23:25.394969 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 25 04:23:25.394969 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 04:23:25.394969 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:25.394969 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:25.394969 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:25.394969 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:25.394969 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:25.394969 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:25.394969 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:25.394969 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:25.394969 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:25.394969 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:25.394969 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:25.394969 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:25.395921 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:25.395921 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:25.395921 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 25 04:23:25.395921 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 25 04:23:25.395921 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 25 04:23:25.395921 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 25 04:23:25.395921 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 25 04:23:25.395921 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 25 04:23:25.395921 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 25 04:23:25.395921 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:25.398723 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 394/800] 10.222.0.25 () {56 vars in 1117 bytes} [Sun Sep 25 04:23:25 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 04:23:25.399743 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28ae5dfa-cc45-4e0c-b3f0-899082f1275d None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:25.399939 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d73ffb03-6aae-42d1-8718-e179eb89d500 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:25.438608 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08b6effc-ea6d-4e54-a550-03ae71d1fa4c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:25.439343 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08b6effc-ea6d-4e54-a550-03ae71d1fa4c None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:25.439719 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08b6effc-ea6d-4e54-a550-03ae71d1fa4c None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:25.440178 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08b6effc-ea6d-4e54-a550-03ae71d1fa4c None admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:25.440935 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08b6effc-ea6d-4e54-a550-03ae71d1fa4c None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:25.441372 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08b6effc-ea6d-4e54-a550-03ae71d1fa4c None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:25.443029 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08b6effc-ea6d-4e54-a550-03ae71d1fa4c None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:25.460724 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 407/801] 10.222.0.25 () {56 vars in 1148 bytes} [Sun Sep 25 04:23:25 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:25.461805 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08b6effc-ea6d-4e54-a550-03ae71d1fa4c None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:25.468676 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:25.534741 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28ae5dfa-cc45-4e0c-b3f0-899082f1275d None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:25.535675 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:25.575979 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bd34507-ba6a-429c-bdff-edba5003b869 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:25.576533 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bd34507-ba6a-429c-bdff-edba5003b869 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:25.576843 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bd34507-ba6a-429c-bdff-edba5003b869 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:25.577114 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bd34507-ba6a-429c-bdff-edba5003b869 None admin] PATH_INFO: `/v3/services/network` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:25.578599 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bd34507-ba6a-429c-bdff-edba5003b869 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:25.580187 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bd34507-ba6a-429c-bdff-edba5003b869 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:25.585407 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08b6effc-ea6d-4e54-a550-03ae71d1fa4c None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:25.589628 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-8bd34507-ba6a-429c-bdff-edba5003b869 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Sep 25 04:23:25.589628 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:25.589628 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 25 04:23:25.589628 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 25 04:23:25.589628 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 25 04:23:25.589628 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 25 04:23:25.589628 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 25 04:23:25.589628 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 04:23:25.589628 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 25 04:23:25.589628 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:25.589628 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 25 04:23:25.589628 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:25.589628 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:25.589628 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:25.589628 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:25.589628 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:25.589628 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:25.590626 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:25.590626 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:25.590626 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:25.590626 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:25.590626 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:25.590626 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:25.590626 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 25 04:23:25.590626 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 25 04:23:25.590626 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 25 04:23:25.590626 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 25 04:23:25.590626 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:25.590626 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:25.590626 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:25.590626 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:25.590626 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:25.590626 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:25.591481 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:25.591481 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:25.591481 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:25.591481 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:25.591481 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:25.591481 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:25.591481 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:25.591481 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:25.591481 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:25.591481 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:25.591481 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 25 04:23:25.591481 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 04:23:25.591481 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 25 04:23:25.591481 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:25.596054 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 395/802] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Sep 25 04:23:25 2022] GET /identity/v3/services/network => generated 88 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:25.597013 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bd34507-ba6a-429c-bdff-edba5003b869 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:25.626301 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bae3d83b-7133-47d0-8cea-2ff4b24395e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:25.627198 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bae3d83b-7133-47d0-8cea-2ff4b24395e2 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:25.627925 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bae3d83b-7133-47d0-8cea-2ff4b24395e2 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:25.628359 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bae3d83b-7133-47d0-8cea-2ff4b24395e2 None admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:25.629319 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bae3d83b-7133-47d0-8cea-2ff4b24395e2 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:25.629632 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bae3d83b-7133-47d0-8cea-2ff4b24395e2 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:25.631143 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bae3d83b-7133-47d0-8cea-2ff4b24395e2 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:25.652461 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 408/803] 10.222.0.25 () {56 vars in 1149 bytes} [Sun Sep 25 04:23:25 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:25.653425 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bae3d83b-7133-47d0-8cea-2ff4b24395e2 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:25.708762 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bd34507-ba6a-429c-bdff-edba5003b869 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:25.723896 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:25.750251 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae228bf7-c54a-4eb0-b31e-baa8386ebccc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:25.750998 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae228bf7-c54a-4eb0-b31e-baa8386ebccc None admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:25.751331 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae228bf7-c54a-4eb0-b31e-baa8386ebccc None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:25.751715 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae228bf7-c54a-4eb0-b31e-baa8386ebccc None admin] PATH_INFO: `/v3/projects/056ca1b1a9974a7581550de0d9217823/users/23ea88f46e7740d295148a93ade29dd9/roles/c51fe7c5ad804040879acfb54269455f` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:25.753580 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae228bf7-c54a-4eb0-b31e-baa8386ebccc None admin] RBAC: Authorizing `identity:create_grant(project_id=056ca1b1a9974a7581550de0d9217823, user_id=23ea88f46e7740d295148a93ade29dd9, role_id=c51fe7c5ad804040879acfb54269455f)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:25.755090 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae228bf7-c54a-4eb0-b31e-baa8386ebccc None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:25.776665 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 396/804] 10.222.0.25 () {58 vars in 1439 bytes} [Sun Sep 25 04:23:25 2022] PUT /identity/v3/projects/056ca1b1a9974a7581550de0d9217823/users/23ea88f46e7740d295148a93ade29dd9/roles/c51fe7c5ad804040879acfb54269455f => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:23:25.776665 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae228bf7-c54a-4eb0-b31e-baa8386ebccc None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:25.784491 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bae3d83b-7133-47d0-8cea-2ff4b24395e2 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:25.820664 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:25.854938 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c38991a3-5e0e-45a5-ad35-0da20d8d550c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:25.855988 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c38991a3-5e0e-45a5-ad35-0da20d8d550c None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:25.856463 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c38991a3-5e0e-45a5-ad35-0da20d8d550c None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:25.856962 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c38991a3-5e0e-45a5-ad35-0da20d8d550c None admin] PATH_INFO: `/v3/services` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:25.857848 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c38991a3-5e0e-45a5-ad35-0da20d8d550c None admin] RBAC: Adding query filter params (name=network) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:25.858348 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c38991a3-5e0e-45a5-ad35-0da20d8d550c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:25.860313 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c38991a3-5e0e-45a5-ad35-0da20d8d550c None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:25.876965 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 409/805] 10.222.0.25 () {56 vars in 1112 bytes} [Sun Sep 25 04:23:25 2022] GET /identity/v3/services?name=network => generated 124 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:25.891226 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c38991a3-5e0e-45a5-ad35-0da20d8d550c None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:25.893514 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae228bf7-c54a-4eb0-b31e-baa8386ebccc None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:25.935144 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35de80d8-6da4-486a-abdc-95c554f37828 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:25.935846 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35de80d8-6da4-486a-abdc-95c554f37828 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:25.936129 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35de80d8-6da4-486a-abdc-95c554f37828 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:25.936407 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35de80d8-6da4-486a-abdc-95c554f37828 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:25.950207 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35de80d8-6da4-486a-abdc-95c554f37828 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:25.952009 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35de80d8-6da4-486a-abdc-95c554f37828 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:25.970130 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-35de80d8-6da4-486a-abdc-95c554f37828 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:23:25.970130 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:25.970130 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:25.970130 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:25.970130 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:25.970130 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:25.970130 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:25.970130 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:25.970130 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:25.970130 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:25.970130 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:25.970130 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:25.970130 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 04:23:25.970130 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 04:23:25.970130 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 04:23:25.970130 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 04:23:25.971155 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:25.971155 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:25.971155 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:25.971155 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:25.971155 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:25.971155 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:25.971155 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:25.971155 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:25.971155 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:25.971155 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:25.971155 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:25.971155 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:25.971155 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:25.971155 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:25.971155 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:25.971155 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:25.971155 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 04:23:25.972082 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 04:23:25.972082 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:23:25.972082 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:25.972082 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 397/806] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:23:25 2022] GET /identity/v3/domains/Default => generated 87 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:25.977261 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35de80d8-6da4-486a-abdc-95c554f37828 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:26.012573 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:26.096137 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c38991a3-5e0e-45a5-ad35-0da20d8d550c None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:26.112885 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35de80d8-6da4-486a-abdc-95c554f37828 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:26.154646 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccdf5121-4bab-49fd-a2f2-e747f3bc30c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:26.155407 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccdf5121-4bab-49fd-a2f2-e747f3bc30c8 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:26.155766 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccdf5121-4bab-49fd-a2f2-e747f3bc30c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:26.156099 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccdf5121-4bab-49fd-a2f2-e747f3bc30c8 None admin] PATH_INFO: `/v3/services` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:26.156767 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccdf5121-4bab-49fd-a2f2-e747f3bc30c8 None admin] RBAC: Adding query filter params (type=network) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:26.157075 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccdf5121-4bab-49fd-a2f2-e747f3bc30c8 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:26.158453 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccdf5121-4bab-49fd-a2f2-e747f3bc30c8 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:26.166308 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ef25ed1-6695-43f6-be19-918862559c45 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:26.166925 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ef25ed1-6695-43f6-be19-918862559c45 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:26.167207 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ef25ed1-6695-43f6-be19-918862559c45 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:26.167481 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ef25ed1-6695-43f6-be19-918862559c45 None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:26.168124 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ef25ed1-6695-43f6-be19-918862559c45 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:26.168428 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ef25ed1-6695-43f6-be19-918862559c45 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:26.169832 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ef25ed1-6695-43f6-be19-918862559c45 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:26.171630 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 410/807] 10.222.0.25 () {56 vars in 1112 bytes} [Sun Sep 25 04:23:25 2022] GET /identity/v3/services?type=network => generated 350 bytes in 280 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:23:26.177315 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccdf5121-4bab-49fd-a2f2-e747f3bc30c8 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:26.190850 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 398/808] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Sep 25 04:23:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:26.196359 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ef25ed1-6695-43f6-be19-918862559c45 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:26.261569 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:26.332573 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccdf5121-4bab-49fd-a2f2-e747f3bc30c8 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:26.372435 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:26.376347 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0174c63b-dedc-48c0-afd5-fb833ca595f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:26.378014 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0174c63b-dedc-48c0-afd5-fb833ca595f9 None admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:26.378583 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0174c63b-dedc-48c0-afd5-fb833ca595f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:26.379049 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0174c63b-dedc-48c0-afd5-fb833ca595f9 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:26.380115 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0174c63b-dedc-48c0-afd5-fb833ca595f9 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:26.381695 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0174c63b-dedc-48c0-afd5-fb833ca595f9 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:26.402821 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ef25ed1-6695-43f6-be19-918862559c45 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:26.419829 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 411/809] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:23:26 2022] POST /identity/v3/endpoints => generated 337 bytes in 243 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 04:23:26.471574 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-866361ba-1638-4852-adfb-331c1f7cf2e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:26.472593 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-866361ba-1638-4852-adfb-331c1f7cf2e7 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:26.473143 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-866361ba-1638-4852-adfb-331c1f7cf2e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:26.473612 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-866361ba-1638-4852-adfb-331c1f7cf2e7 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:26.495883 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-866361ba-1638-4852-adfb-331c1f7cf2e7 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:26.497444 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-866361ba-1638-4852-adfb-331c1f7cf2e7 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:26.511061 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-866361ba-1638-4852-adfb-331c1f7cf2e7 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 04:23:26.511061 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:26.511061 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:26.511061 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:26.511061 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:26.511061 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:26.511061 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:26.511061 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:26.511061 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:26.511061 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:26.511061 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:26.511061 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:26.511061 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 25 04:23:26.511061 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 25 04:23:26.511061 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 25 04:23:26.512085 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 25 04:23:26.512085 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:26.512085 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:26.512085 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:26.512085 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:26.512085 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:26.512085 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:26.512085 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:26.512085 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:26.512085 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:26.512085 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:26.512085 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:26.512085 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:26.512085 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:26.512085 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:26.512085 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:26.512085 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:26.513032 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 25 04:23:26.513032 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 25 04:23:26.513032 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 25 04:23:26.513032 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 25 04:23:26.513032 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 25 04:23:26.513032 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 25 04:23:26.513032 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 04:23:26.513032 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:26.513032 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 399/810] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Sep 25 04:23:26 2022] GET /identity/v3/projects/service => generated 88 bytes in 316 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 04:23:26.518537 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0174c63b-dedc-48c0-afd5-fb833ca595f9 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:26.655814 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:26.663042 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0174c63b-dedc-48c0-afd5-fb833ca595f9 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:26.713802 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc60b82e-942c-4f82-abe2-6337e51e5801 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:26.714429 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc60b82e-942c-4f82-abe2-6337e51e5801 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:26.714709 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc60b82e-942c-4f82-abe2-6337e51e5801 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:26.714983 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc60b82e-942c-4f82-abe2-6337e51e5801 None admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:26.715684 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc60b82e-942c-4f82-abe2-6337e51e5801 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:26.715991 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc60b82e-942c-4f82-abe2-6337e51e5801 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:26.717440 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc60b82e-942c-4f82-abe2-6337e51e5801 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:26.739324 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 412/811] 10.222.0.25 () {56 vars in 1148 bytes} [Sun Sep 25 04:23:26 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:26.745056 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-866361ba-1638-4852-adfb-331c1f7cf2e7 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:26.867441 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-866361ba-1638-4852-adfb-331c1f7cf2e7 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:26.903190 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ca5c79c-5d29-4e0e-871d-5dcc6a88258a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:26.903861 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca5c79c-5d29-4e0e-871d-5dcc6a88258a None admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:26.904143 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca5c79c-5d29-4e0e-871d-5dcc6a88258a None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:26.904414 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca5c79c-5d29-4e0e-871d-5dcc6a88258a None admin] PATH_INFO: `/v3/projects/056ca1b1a9974a7581550de0d9217823/users/050b3aab44bb4b17aa4a1ba7cccdb231/roles/9b44520f482c4423bd10568d0f86718c` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:26.916256 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:26.922255 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ca5c79c-5d29-4e0e-871d-5dcc6a88258a None admin] RBAC: Authorizing `identity:create_grant(project_id=056ca1b1a9974a7581550de0d9217823, user_id=050b3aab44bb4b17aa4a1ba7cccdb231, role_id=9b44520f482c4423bd10568d0f86718c)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:26.923748 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ca5c79c-5d29-4e0e-871d-5dcc6a88258a None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:26.939770 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 400/812] 10.222.0.25 () {58 vars in 1439 bytes} [Sun Sep 25 04:23:26 2022] PUT /identity/v3/projects/056ca1b1a9974a7581550de0d9217823/users/050b3aab44bb4b17aa4a1ba7cccdb231/roles/9b44520f482c4423bd10568d0f86718c => generated 0 bytes in 195 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:23:27.130089 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4281520e-401f-43a2-8597-48604220aca2 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:27.130089 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4281520e-401f-43a2-8597-48604220aca2 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:27.130089 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4281520e-401f-43a2-8597-48604220aca2 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:27.131189 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 413/813] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:23:27 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:23:27.136918 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6aef792-27ab-4e32-a899-6de63cd73c9d None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:27.137212 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6aef792-27ab-4e32-a899-6de63cd73c9d None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:27.137488 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6aef792-27ab-4e32-a899-6de63cd73c9d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:27.160055 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-f6aef792-27ab-4e32-a899-6de63cd73c9d None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:23:27.161311 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-f6aef792-27ab-4e32-a899-6de63cd73c9d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:23:27.242861 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 401/814] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:23:27 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 04:23:27.372766 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d61cab30-a0c9-4d67-875e-2fec5afbf91c None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:27.373124 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d61cab30-a0c9-4d67-875e-2fec5afbf91c None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:27.373319 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d61cab30-a0c9-4d67-875e-2fec5afbf91c None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:27.375652 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 414/815] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:23:27 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:23:27.382467 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c93cdf-8151-4021-937e-bf90b7788386 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:27.382927 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c93cdf-8151-4021-937e-bf90b7788386 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:27.383098 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c93cdf-8151-4021-937e-bf90b7788386 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:27.403850 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-95c93cdf-8151-4021-937e-bf90b7788386 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:23:27.406099 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-95c93cdf-8151-4021-937e-bf90b7788386 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:23:27.492201 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 402/816] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:23:27 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 04:23:27.541229 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d61cab30-a0c9-4d67-875e-2fec5afbf91c None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:27.647723 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d61cab30-a0c9-4d67-875e-2fec5afbf91c None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:27.684155 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b01ea89f-c90f-4742-981f-4d91fdde2b13 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:27.685198 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b01ea89f-c90f-4742-981f-4d91fdde2b13 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:27.685723 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b01ea89f-c90f-4742-981f-4d91fdde2b13 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:27.686212 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b01ea89f-c90f-4742-981f-4d91fdde2b13 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:27.697903 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b01ea89f-c90f-4742-981f-4d91fdde2b13 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 04:23:27.704223 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-b01ea89f-c90f-4742-981f-4d91fdde2b13 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 04:23:27.704223 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:27.704223 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:27.704223 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:27.704223 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:27.704223 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:27.704223 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:27.704223 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:27.704223 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:27.704223 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:27.704223 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:27.704223 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:27.704223 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 04:23:27.704223 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 04:23:27.704223 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 04:23:27.704223 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 04:23:27.705204 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 04:23:27.705204 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 04:23:27.705204 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 04:23:27.705204 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 04:23:27.705204 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:27.705204 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:27.705204 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:27.705204 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:27.705204 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:27.705204 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:27.705204 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:27.705204 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:27.705204 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:27.705204 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:27.705204 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:27.705204 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:27.706053 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:27.706053 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:27.706053 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:27.706053 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:27.706053 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 04:23:27.706053 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 04:23:27.706053 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 04:23:27.706053 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 04:23:27.706053 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 04:23:27.706053 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 04:23:27.706053 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 04:23:27.706053 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:23:27.708695 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 415/817] 10.222.0.25 () {56 vars in 1096 bytes} [Sun Sep 25 04:23:27 2022] GET /identity/v3/roles/admin => generated 83 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:27.716668 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95c93cdf-8151-4021-937e-bf90b7788386 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:27.768251 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:27.783531 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b01ea89f-c90f-4742-981f-4d91fdde2b13 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:27.811092 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:27.822252 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95c93cdf-8151-4021-937e-bf90b7788386 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:27.860852 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56ce4725-6246-413c-8b3f-183846ac5305 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:27.861462 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ce4725-6246-413c-8b3f-183846ac5305 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:27.861737 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ce4725-6246-413c-8b3f-183846ac5305 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:27.862006 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ce4725-6246-413c-8b3f-183846ac5305 None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:27.862726 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56ce4725-6246-413c-8b3f-183846ac5305 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:27.863032 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56ce4725-6246-413c-8b3f-183846ac5305 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:27.864404 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56ce4725-6246-413c-8b3f-183846ac5305 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:27.873629 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 403/818] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Sep 25 04:23:27 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:27.878848 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56ce4725-6246-413c-8b3f-183846ac5305 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:27.908177 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b01ea89f-c90f-4742-981f-4d91fdde2b13 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:27.948749 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f068dbe-f5eb-4f62-b976-59bf61093219 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:27.949656 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f068dbe-f5eb-4f62-b976-59bf61093219 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:27.950081 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f068dbe-f5eb-4f62-b976-59bf61093219 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:27.950496 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f068dbe-f5eb-4f62-b976-59bf61093219 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:27.951463 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f068dbe-f5eb-4f62-b976-59bf61093219 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:27.953091 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f068dbe-f5eb-4f62-b976-59bf61093219 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:27.969217 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-1f068dbe-f5eb-4f62-b976-59bf61093219 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 25 04:23:27.969217 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:27.969217 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 25 04:23:27.969217 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 25 04:23:27.969217 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 25 04:23:27.969217 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 25 04:23:27.969217 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 25 04:23:27.969217 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 04:23:27.969217 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 25 04:23:27.969217 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:23:27.969217 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 25 04:23:27.969217 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:23:27.969217 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:27.969217 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:27.969217 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:27.969217 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:27.969217 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:27.970450 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:27.970450 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:27.970450 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:27.970450 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:27.970450 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:27.970450 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:27.970450 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 25 04:23:27.970450 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 25 04:23:27.970450 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 25 04:23:27.970450 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 25 04:23:27.970450 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:27.970450 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:27.970450 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:27.970450 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:27.970450 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:27.970450 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:27.971503 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:27.971503 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:27.971503 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:27.971503 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:27.971503 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:27.971503 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:27.971503 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:27.971503 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:27.971503 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:27.971503 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:27.971503 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 25 04:23:27.971503 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 04:23:27.971503 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 25 04:23:27.971503 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:23:27.978853 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 416/819] 10.222.0.25 () {56 vars in 1129 bytes} [Sun Sep 25 04:23:27 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 04:23:27.979864 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f068dbe-f5eb-4f62-b976-59bf61093219 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:27.994196 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56ce4725-6246-413c-8b3f-183846ac5305 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:28.007998 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:28.031659 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee571aad-2b8c-4799-bfc4-9588dcee0875 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:28.032386 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee571aad-2b8c-4799-bfc4-9588dcee0875 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:28.032764 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee571aad-2b8c-4799-bfc4-9588dcee0875 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:28.033122 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee571aad-2b8c-4799-bfc4-9588dcee0875 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:28.043900 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee571aad-2b8c-4799-bfc4-9588dcee0875 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:28.045631 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee571aad-2b8c-4799-bfc4-9588dcee0875 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:28.062303 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-ee571aad-2b8c-4799-bfc4-9588dcee0875 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:23:28.062303 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:28.062303 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:28.062303 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:28.062303 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:28.062303 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:28.062303 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:28.062303 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:28.062303 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:28.062303 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:28.062303 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:28.062303 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:28.062303 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 04:23:28.062303 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 04:23:28.062303 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 04:23:28.062303 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 04:23:28.064235 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:28.064235 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:28.064235 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:28.064235 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:28.064235 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:28.064235 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:28.064235 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:28.064235 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:28.064235 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:28.064235 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:28.064235 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:28.064235 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:28.064235 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:28.064235 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:28.064235 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:28.064235 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:28.064235 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 04:23:28.065200 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 04:23:28.065200 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:23:28.065200 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:28.071105 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 404/820] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:23:27 2022] GET /identity/v3/domains/Default => generated 87 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:28.072751 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee571aad-2b8c-4799-bfc4-9588dcee0875 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:28.091498 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:28.114595 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f068dbe-f5eb-4f62-b976-59bf61093219 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:28.153787 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8826cbc9-102d-4a3d-9ec2-50c7e9f9d76d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:28.155075 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8826cbc9-102d-4a3d-9ec2-50c7e9f9d76d None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:28.156141 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8826cbc9-102d-4a3d-9ec2-50c7e9f9d76d None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:28.156546 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8826cbc9-102d-4a3d-9ec2-50c7e9f9d76d None admin] PATH_INFO: `/v3/services` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:28.157377 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8826cbc9-102d-4a3d-9ec2-50c7e9f9d76d None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:28.157843 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8826cbc9-102d-4a3d-9ec2-50c7e9f9d76d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:28.159330 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8826cbc9-102d-4a3d-9ec2-50c7e9f9d76d None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:28.169823 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 417/821] 10.222.0.25 () {56 vars in 1124 bytes} [Sun Sep 25 04:23:27 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:28.170980 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8826cbc9-102d-4a3d-9ec2-50c7e9f9d76d None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:28.182588 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee571aad-2b8c-4799-bfc4-9588dcee0875 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:28.221526 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84884ef1-3310-400f-b1ff-0e4886ecd3f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:28.222330 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84884ef1-3310-400f-b1ff-0e4886ecd3f5 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:28.222697 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84884ef1-3310-400f-b1ff-0e4886ecd3f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:28.223371 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84884ef1-3310-400f-b1ff-0e4886ecd3f5 None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:28.223783 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84884ef1-3310-400f-b1ff-0e4886ecd3f5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:28.224168 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84884ef1-3310-400f-b1ff-0e4886ecd3f5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:28.225713 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84884ef1-3310-400f-b1ff-0e4886ecd3f5 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:28.244832 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 405/822] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Sep 25 04:23:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:23:28.246618 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84884ef1-3310-400f-b1ff-0e4886ecd3f5 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:28.274407 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:28.304319 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8826cbc9-102d-4a3d-9ec2-50c7e9f9d76d None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:28.342479 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f1223ec-56c2-47d1-826c-3e7cc138cd4a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:28.343196 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f1223ec-56c2-47d1-826c-3e7cc138cd4a None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:28.343544 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f1223ec-56c2-47d1-826c-3e7cc138cd4a None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:28.343926 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f1223ec-56c2-47d1-826c-3e7cc138cd4a None admin] PATH_INFO: `/v3/services` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:28.344592 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f1223ec-56c2-47d1-826c-3e7cc138cd4a None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:28.344965 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f1223ec-56c2-47d1-826c-3e7cc138cd4a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:28.346307 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f1223ec-56c2-47d1-826c-3e7cc138cd4a None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:28.351800 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 418/823] 10.222.0.25 () {56 vars in 1124 bytes} [Sun Sep 25 04:23:28 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:28.356438 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c3a0088-163a-46af-a8a9-d26c01f71e05 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:28.356438 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c3a0088-163a-46af-a8a9-d26c01f71e05 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:28.356438 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c3a0088-163a-46af-a8a9-d26c01f71e05 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:28.356438 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 419/824] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:23:28 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:23:28.356826 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84884ef1-3310-400f-b1ff-0e4886ecd3f5 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:28.357295 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c3a0088-163a-46af-a8a9-d26c01f71e05 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:28.373352 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:28.398696 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51db0cd1-b5c5-49a5-b65a-76747aa1789f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:28.399330 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51db0cd1-b5c5-49a5-b65a-76747aa1789f None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:28.399611 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51db0cd1-b5c5-49a5-b65a-76747aa1789f None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:28.399884 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51db0cd1-b5c5-49a5-b65a-76747aa1789f None admin] PATH_INFO: `/v3/users/nova` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:28.407763 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51db0cd1-b5c5-49a5-b65a-76747aa1789f None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:28.409242 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51db0cd1-b5c5-49a5-b65a-76747aa1789f None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:28.421786 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-51db0cd1-b5c5-49a5-b65a-76747aa1789f None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 25 04:23:28.421786 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:28.421786 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:28.421786 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:28.421786 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:28.421786 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:28.421786 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:28.421786 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:28.421786 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:28.421786 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:28.421786 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:28.421786 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:28.421786 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 25 04:23:28.421786 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 25 04:23:28.421786 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 25 04:23:28.421786 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 25 04:23:28.422756 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:28.422756 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:28.422756 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 25 04:23:28.422756 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 04:23:28.422756 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 25 04:23:28.422756 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 04:23:28.422756 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:28.422756 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:28.422756 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:28.422756 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:28.422756 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:28.422756 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:28.422756 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:28.422756 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:28.422756 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:28.422756 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:28.422756 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:28.422756 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:28.423718 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:28.423718 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:28.423718 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 25 04:23:28.423718 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 25 04:23:28.423718 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 25 04:23:28.423718 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 25 04:23:28.423718 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 25 04:23:28.423718 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 25 04:23:28.423718 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 25 04:23:28.423718 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:28.423718 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 406/825] 10.222.0.25 () {56 vars in 1093 bytes} [Sun Sep 25 04:23:28 2022] GET /identity/v3/users/nova => generated 82 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:28.424318 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95f64db7-e6e0-405b-9902-0486aefb6123 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:28.424401 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95f64db7-e6e0-405b-9902-0486aefb6123 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:28.424846 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95f64db7-e6e0-405b-9902-0486aefb6123 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:28.443088 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-95f64db7-e6e0-405b-9902-0486aefb6123 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:23:28.444286 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-95f64db7-e6e0-405b-9902-0486aefb6123 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:23:28.481893 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c3a0088-163a-46af-a8a9-d26c01f71e05 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:28.518333 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84d0e2f5-8968-4a7f-a357-6649553b523a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:28.518943 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84d0e2f5-8968-4a7f-a357-6649553b523a None admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:28.519223 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84d0e2f5-8968-4a7f-a357-6649553b523a None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:28.519601 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84d0e2f5-8968-4a7f-a357-6649553b523a None admin] PATH_INFO: `/v3/endpoints` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:28.520189 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84d0e2f5-8968-4a7f-a357-6649553b523a None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:28.521589 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84d0e2f5-8968-4a7f-a357-6649553b523a None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:28.523401 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 407/826] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:23:28 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 04:23:28.524079 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95f64db7-e6e0-405b-9902-0486aefb6123 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:28.540441 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:28.565110 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 420/827] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:23:28 2022] POST /identity/v3/endpoints => generated 346 bytes in 208 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:23:28.649009 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95f64db7-e6e0-405b-9902-0486aefb6123 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:28.696389 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c45b2fb-7904-4797-a8c3-1e40f0601113 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:28.697399 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c45b2fb-7904-4797-a8c3-1e40f0601113 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:28.697930 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c45b2fb-7904-4797-a8c3-1e40f0601113 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:28.698427 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c45b2fb-7904-4797-a8c3-1e40f0601113 None admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:28.699425 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c45b2fb-7904-4797-a8c3-1e40f0601113 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:28.699935 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c45b2fb-7904-4797-a8c3-1e40f0601113 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:28.701890 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c45b2fb-7904-4797-a8c3-1e40f0601113 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:28.723404 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 408/828] 10.222.0.25 () {56 vars in 1133 bytes} [Sun Sep 25 04:23:28 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:28.731642 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84d0e2f5-8968-4a7f-a357-6649553b523a None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:28.803635 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c45b2fb-7904-4797-a8c3-1e40f0601113 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:28.806637 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:28.847559 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84d0e2f5-8968-4a7f-a357-6649553b523a None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:28.876563 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:28.886134 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77690131-5e7c-4569-bb25-4574fe255506 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:28.887144 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77690131-5e7c-4569-bb25-4574fe255506 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:28.887574 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77690131-5e7c-4569-bb25-4574fe255506 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:28.887993 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77690131-5e7c-4569-bb25-4574fe255506 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:28.915525 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c45b2fb-7904-4797-a8c3-1e40f0601113 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:28.962356 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68297430-e99b-45e6-bac1-8c77417c699c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:28.963244 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68297430-e99b-45e6-bac1-8c77417c699c None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:28.963939 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68297430-e99b-45e6-bac1-8c77417c699c None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:28.964213 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68297430-e99b-45e6-bac1-8c77417c699c None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:28.977448 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77690131-5e7c-4569-bb25-4574fe255506 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:28.978788 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68297430-e99b-45e6-bac1-8c77417c699c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 04:23:28.978996 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77690131-5e7c-4569-bb25-4574fe255506 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:28.983978 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-68297430-e99b-45e6-bac1-8c77417c699c None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 04:23:28.983978 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:28.983978 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:28.983978 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:28.983978 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:28.983978 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:28.983978 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:28.983978 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:28.983978 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:28.983978 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:28.983978 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:28.983978 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:28.983978 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 04:23:28.983978 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 04:23:28.983978 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 04:23:28.983978 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 04:23:28.985094 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 04:23:28.985094 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 04:23:28.985094 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 04:23:28.985094 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 04:23:28.985094 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:28.985094 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:28.985094 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:28.985094 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:28.985094 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:28.985094 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:28.985094 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:28.985094 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:28.985094 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:28.985094 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:28.985094 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:28.985094 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:28.986344 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:28.986344 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:28.986344 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:28.986344 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:28.986344 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 04:23:28.986344 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 04:23:28.986344 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 04:23:28.986344 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 04:23:28.986344 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 04:23:28.986344 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 04:23:28.986344 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 04:23:28.986344 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:28.986344 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 409/829] 10.222.0.25 () {56 vars in 1096 bytes} [Sun Sep 25 04:23:28 2022] GET /identity/v3/roles/admin => generated 83 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 04:23:28.992720 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68297430-e99b-45e6-bac1-8c77417c699c None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:29.053890 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:29.093589 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 421/830] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Sep 25 04:23:28 2022] GET /identity/v3/auth/tokens => generated 1161 bytes in 365 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 25 04:23:29.099877 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77690131-5e7c-4569-bb25-4574fe255506 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:29.106238 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68297430-e99b-45e6-bac1-8c77417c699c None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:29.141686 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:29.152085 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef3af17f-6b54-4d96-aee6-5627b3b8ace8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:29.153245 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef3af17f-6b54-4d96-aee6-5627b3b8ace8 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:29.153799 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef3af17f-6b54-4d96-aee6-5627b3b8ace8 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:29.154274 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef3af17f-6b54-4d96-aee6-5627b3b8ace8 None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:29.155141 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef3af17f-6b54-4d96-aee6-5627b3b8ace8 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:29.155724 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef3af17f-6b54-4d96-aee6-5627b3b8ace8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:29.157626 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef3af17f-6b54-4d96-aee6-5627b3b8ace8 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:29.167631 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 410/831] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Sep 25 04:23:28 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:29.173518 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef3af17f-6b54-4d96-aee6-5627b3b8ace8 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:29.220555 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77690131-5e7c-4569-bb25-4574fe255506 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:29.258805 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab8d76d5-9248-4df2-b8a8-db45a802fd2f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:29.259762 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab8d76d5-9248-4df2-b8a8-db45a802fd2f None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:29.260226 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab8d76d5-9248-4df2-b8a8-db45a802fd2f None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:29.260808 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab8d76d5-9248-4df2-b8a8-db45a802fd2f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:29.271864 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab8d76d5-9248-4df2-b8a8-db45a802fd2f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:29.273779 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab8d76d5-9248-4df2-b8a8-db45a802fd2f None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:29.289365 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-ab8d76d5-9248-4df2-b8a8-db45a802fd2f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:23:29.289365 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:29.289365 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:29.289365 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:29.289365 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:29.289365 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:29.289365 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:29.289365 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:29.289365 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:29.289365 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:29.289365 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:29.289365 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:29.289365 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 04:23:29.289365 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 04:23:29.289365 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 04:23:29.289365 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 04:23:29.290347 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:29.290347 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:29.290347 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:29.290347 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:29.290347 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:29.290347 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:29.290347 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:29.290347 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:29.290347 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:29.290347 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:29.290347 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:29.290347 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:29.290347 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:29.290347 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:29.290347 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:29.290347 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:29.290347 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 04:23:29.291261 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 04:23:29.291261 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:23:29.291261 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:23:29.291433 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef3af17f-6b54-4d96-aee6-5627b3b8ace8 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:29.292896 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 422/832] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:23:29 2022] GET /identity/v3/domains/Default => generated 87 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:29.298280 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb031d12-7212-40e7-8b1e-a93292bfdfad None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:29.298482 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb031d12-7212-40e7-8b1e-a93292bfdfad None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:29.298719 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb031d12-7212-40e7-8b1e-a93292bfdfad None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:29.299992 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 423/833] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:23:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:23:29.301239 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb031d12-7212-40e7-8b1e-a93292bfdfad None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:29.327987 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:29.328960 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ef124e1-b238-4f8d-b7d3-d704353c1a5d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:29.329601 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ef124e1-b238-4f8d-b7d3-d704353c1a5d None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:29.329980 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ef124e1-b238-4f8d-b7d3-d704353c1a5d None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:29.330246 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ef124e1-b238-4f8d-b7d3-d704353c1a5d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:29.341623 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ef124e1-b238-4f8d-b7d3-d704353c1a5d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:29.343274 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ef124e1-b238-4f8d-b7d3-d704353c1a5d None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:29.358274 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-1ef124e1-b238-4f8d-b7d3-d704353c1a5d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:23:29.358274 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:29.358274 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:29.358274 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:29.358274 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:29.358274 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:29.358274 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:29.358274 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:29.358274 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:29.358274 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:29.358274 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:29.358274 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:29.358274 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 04:23:29.358274 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 04:23:29.358274 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 04:23:29.358274 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 04:23:29.360178 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:29.360178 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:29.360178 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:29.360178 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:29.360178 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:29.360178 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:29.360178 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:29.360178 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:29.360178 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:29.360178 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:29.360178 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:29.360178 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:29.360178 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:29.360178 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:29.360178 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:29.360178 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:29.360178 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 04:23:29.361154 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 04:23:29.361154 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:23:29.361154 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:29.364617 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 411/834] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:23:29 2022] GET /identity/v3/domains/Default => generated 87 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 04:23:29.365903 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a52f9bd4-9cbd-4be3-aa43-16c07eee1762 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:29.366189 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a52f9bd4-9cbd-4be3-aa43-16c07eee1762 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:29.366464 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a52f9bd4-9cbd-4be3-aa43-16c07eee1762 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:29.384974 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-a52f9bd4-9cbd-4be3-aa43-16c07eee1762 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:23:29.385522 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-a52f9bd4-9cbd-4be3-aa43-16c07eee1762 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:23:29.418051 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb031d12-7212-40e7-8b1e-a93292bfdfad None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:29.425091 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:29.455394 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bda4739b-e2ee-4cc7-9a9a-9622bef51bc9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:29.456013 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bda4739b-e2ee-4cc7-9a9a-9622bef51bc9 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:29.456285 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bda4739b-e2ee-4cc7-9a9a-9622bef51bc9 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:29.456552 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bda4739b-e2ee-4cc7-9a9a-9622bef51bc9 None admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:29.457235 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bda4739b-e2ee-4cc7-9a9a-9622bef51bc9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:29.457535 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bda4739b-e2ee-4cc7-9a9a-9622bef51bc9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:29.458892 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bda4739b-e2ee-4cc7-9a9a-9622bef51bc9 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:29.461439 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 412/835] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:23:29 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 04:23:29.462104 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a52f9bd4-9cbd-4be3-aa43-16c07eee1762 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:29.488611 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 424/836] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Sep 25 04:23:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:29.493415 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bda4739b-e2ee-4cc7-9a9a-9622bef51bc9 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:29.590193 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a52f9bd4-9cbd-4be3-aa43-16c07eee1762 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:29.594281 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:29.615860 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bda4739b-e2ee-4cc7-9a9a-9622bef51bc9 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:29.639973 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c361fd66-6312-4c14-af55-ee545dc5654b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:29.641350 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c361fd66-6312-4c14-af55-ee545dc5654b None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:29.641705 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c361fd66-6312-4c14-af55-ee545dc5654b None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:29.642049 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c361fd66-6312-4c14-af55-ee545dc5654b None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:29.642758 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c361fd66-6312-4c14-af55-ee545dc5654b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:29.643136 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c361fd66-6312-4c14-af55-ee545dc5654b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:29.644597 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c361fd66-6312-4c14-af55-ee545dc5654b None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:29.655041 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e995dd0-18cc-46b9-b5aa-a021337c0650 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:29.656845 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e995dd0-18cc-46b9-b5aa-a021337c0650 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:29.657350 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e995dd0-18cc-46b9-b5aa-a021337c0650 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:29.657749 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 413/837] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Sep 25 04:23:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:29.657970 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e995dd0-18cc-46b9-b5aa-a021337c0650 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:29.662769 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c361fd66-6312-4c14-af55-ee545dc5654b None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:29.668621 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e995dd0-18cc-46b9-b5aa-a021337c0650 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:29.669804 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e995dd0-18cc-46b9-b5aa-a021337c0650 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:29.681995 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-1e995dd0-18cc-46b9-b5aa-a021337c0650 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 04:23:29.681995 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:29.681995 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:29.681995 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:29.681995 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:29.681995 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:29.681995 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:29.681995 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:29.681995 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:29.681995 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:29.681995 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:29.681995 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:29.681995 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 25 04:23:29.681995 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 25 04:23:29.681995 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 25 04:23:29.682933 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 25 04:23:29.682933 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:29.682933 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:29.682933 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:29.682933 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:29.682933 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:29.682933 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:29.682933 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:29.682933 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:29.682933 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:29.682933 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:29.682933 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:29.682933 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:29.682933 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:29.682933 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:29.682933 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:29.682933 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:29.683859 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 25 04:23:29.683859 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 25 04:23:29.683859 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 25 04:23:29.683859 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 25 04:23:29.683859 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 25 04:23:29.683859 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 25 04:23:29.683859 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 04:23:29.683859 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:23:29.688263 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 425/838] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Sep 25 04:23:29 2022] GET /identity/v3/projects/service => generated 88 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:29.689150 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e995dd0-18cc-46b9-b5aa-a021337c0650 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:29.709273 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:29.791471 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c361fd66-6312-4c14-af55-ee545dc5654b None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:29.797989 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e995dd0-18cc-46b9-b5aa-a021337c0650 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:29.826150 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-943f0f70-73c0-4a56-8c98-932ef49d21b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:29.826787 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-943f0f70-73c0-4a56-8c98-932ef49d21b8 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:29.827576 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-943f0f70-73c0-4a56-8c98-932ef49d21b8 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:29.827850 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-943f0f70-73c0-4a56-8c98-932ef49d21b8 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:29.835120 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fcb8466-f224-4c7c-a585-acbde71f670d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:29.835730 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fcb8466-f224-4c7c-a585-acbde71f670d None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:29.836009 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fcb8466-f224-4c7c-a585-acbde71f670d None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:29.836109 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-943f0f70-73c0-4a56-8c98-932ef49d21b8 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:29.836274 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fcb8466-f224-4c7c-a585-acbde71f670d None admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:29.836994 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fcb8466-f224-4c7c-a585-acbde71f670d None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:29.837304 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fcb8466-f224-4c7c-a585-acbde71f670d None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:29.837477 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-943f0f70-73c0-4a56-8c98-932ef49d21b8 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:29.838707 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fcb8466-f224-4c7c-a585-acbde71f670d None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:29.848542 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:29.848993 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-943f0f70-73c0-4a56-8c98-932ef49d21b8 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 25 04:23:29.848993 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:29.848993 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:29.848993 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:29.848993 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:29.848993 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:29.848993 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:29.848993 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:29.848993 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:29.848993 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:29.848993 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:29.848993 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:29.848993 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 25 04:23:29.848993 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 25 04:23:29.848993 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 25 04:23:29.848993 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 25 04:23:29.849944 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:29.849944 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:29.849944 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 25 04:23:29.849944 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 04:23:29.849944 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 25 04:23:29.849944 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 04:23:29.849944 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:29.849944 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:29.849944 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:29.849944 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:29.849944 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:29.849944 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:29.849944 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:29.849944 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:29.849944 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:29.849944 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:29.849944 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:29.849944 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:29.850899 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:29.850899 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:29.850899 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 25 04:23:29.850899 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 25 04:23:29.850899 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 25 04:23:29.850899 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 25 04:23:29.850899 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 25 04:23:29.850899 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 25 04:23:29.850899 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 25 04:23:29.850899 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:29.850899 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 414/839] 10.222.0.25 () {56 vars in 1096 bytes} [Sun Sep 25 04:23:29 2022] GET /identity/v3/users/swift => generated 83 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 04:23:29.850899 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-943f0f70-73c0-4a56-8c98-932ef49d21b8 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:29.863700 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 426/840] 10.222.0.25 () {56 vars in 1148 bytes} [Sun Sep 25 04:23:29 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:29.864579 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fcb8466-f224-4c7c-a585-acbde71f670d None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:29.962294 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:29.973324 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fcb8466-f224-4c7c-a585-acbde71f670d None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:29.973469 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-943f0f70-73c0-4a56-8c98-932ef49d21b8 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:30.009176 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7caed48b-d526-40fe-b34a-2e6d632adad4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:30.009809 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7caed48b-d526-40fe-b34a-2e6d632adad4 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:30.010089 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7caed48b-d526-40fe-b34a-2e6d632adad4 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:30.010360 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7caed48b-d526-40fe-b34a-2e6d632adad4 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:30.011177 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7af0f970-14ab-40ce-afe4-d752e265415b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:30.011775 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7af0f970-14ab-40ce-afe4-d752e265415b None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:30.012153 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7af0f970-14ab-40ce-afe4-d752e265415b None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:30.012425 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7af0f970-14ab-40ce-afe4-d752e265415b None admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:30.013177 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7af0f970-14ab-40ce-afe4-d752e265415b None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:30.013485 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7af0f970-14ab-40ce-afe4-d752e265415b None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:30.015407 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7af0f970-14ab-40ce-afe4-d752e265415b None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:30.021058 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7caed48b-d526-40fe-b34a-2e6d632adad4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 04:23:30.026277 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-7caed48b-d526-40fe-b34a-2e6d632adad4 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 25 04:23:30.026277 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:30.026277 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:30.026277 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:30.026277 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:30.026277 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:30.026277 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:30.026277 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:30.026277 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:30.026277 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:30.026277 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:30.026277 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:30.026277 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 04:23:30.026277 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 04:23:30.026277 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 04:23:30.026277 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 04:23:30.027205 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 04:23:30.027205 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 04:23:30.027205 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 04:23:30.027205 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 04:23:30.027205 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:30.027205 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:30.027205 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:30.027205 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:30.027205 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:30.027205 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:30.027205 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:30.027205 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:30.027205 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:30.027205 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:30.027205 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:30.027205 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:30.028092 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:30.028092 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:30.028092 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:30.028092 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:30.028092 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 04:23:30.028092 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 04:23:30.028092 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 04:23:30.028092 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 04:23:30.028092 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 04:23:30.028092 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 04:23:30.028092 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 25 04:23:30.028092 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:30.028092 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 415/841] 10.222.0.25 () {56 vars in 1102 bytes} [Sun Sep 25 04:23:29 2022] GET /identity/v3/roles/service => generated 85 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:30.028849 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7caed48b-d526-40fe-b34a-2e6d632adad4 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:30.033960 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 427/842] 10.222.0.25 () {56 vars in 1135 bytes} [Sun Sep 25 04:23:29 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:30.039251 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7af0f970-14ab-40ce-afe4-d752e265415b None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:30.102088 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:30.131724 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7caed48b-d526-40fe-b34a-2e6d632adad4 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:30.148493 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7af0f970-14ab-40ce-afe4-d752e265415b None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:30.166560 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c04e9953-a2a0-4285-8f29-1f1eb3118267 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:30.167234 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c04e9953-a2a0-4285-8f29-1f1eb3118267 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:30.167582 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c04e9953-a2a0-4285-8f29-1f1eb3118267 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:30.167921 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c04e9953-a2a0-4285-8f29-1f1eb3118267 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:30.168734 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c04e9953-a2a0-4285-8f29-1f1eb3118267 None admin] RBAC: Adding query filter params (role.id=c51fe7c5ad804040879acfb54269455f, scope.project.id=056ca1b1a9974a7581550de0d9217823, user.id=e703b42c53974daaa608b0ccc3f018b1) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:30.169109 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c04e9953-a2a0-4285-8f29-1f1eb3118267 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:30.170583 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c04e9953-a2a0-4285-8f29-1f1eb3118267 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:30.184665 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 416/843] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Sep 25 04:23:30 2022] GET /identity/v3/role_assignments?user.id=e703b42c53974daaa608b0ccc3f018b1&scope.project.id=056ca1b1a9974a7581550de0d9217823&role.id=c51fe7c5ad804040879acfb54269455f => generated 608 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:30.184665 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c04e9953-a2a0-4285-8f29-1f1eb3118267 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:30.197746 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-147e30b8-b9b8-45ba-b28a-5d116e04a539 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:30.198502 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-147e30b8-b9b8-45ba-b28a-5d116e04a539 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:30.198855 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-147e30b8-b9b8-45ba-b28a-5d116e04a539 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:30.199603 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-147e30b8-b9b8-45ba-b28a-5d116e04a539 None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:30.200976 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-147e30b8-b9b8-45ba-b28a-5d116e04a539 None admin] RBAC: Adding query filter params (name=service) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:30.201356 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-147e30b8-b9b8-45ba-b28a-5d116e04a539 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:30.203112 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-147e30b8-b9b8-45ba-b28a-5d116e04a539 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:30.207017 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:30.221001 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 428/844] 10.222.0.25 () {56 vars in 1103 bytes} [Sun Sep 25 04:23:30 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:30.222730 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-147e30b8-b9b8-45ba-b28a-5d116e04a539 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:30.310579 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c04e9953-a2a0-4285-8f29-1f1eb3118267 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:30.350968 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4af6dd23-4f2f-4eb9-abc2-759306c08bdb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:30.351737 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af6dd23-4f2f-4eb9-abc2-759306c08bdb None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:30.352135 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af6dd23-4f2f-4eb9-abc2-759306c08bdb None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:30.352571 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af6dd23-4f2f-4eb9-abc2-759306c08bdb None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:30.365281 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-147e30b8-b9b8-45ba-b28a-5d116e04a539 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:30.393461 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:30.414370 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c264dc1a-907b-4104-9f42-e103b4d1519b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:30.415028 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c264dc1a-907b-4104-9f42-e103b4d1519b None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:30.415289 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c264dc1a-907b-4104-9f42-e103b4d1519b None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:30.415612 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c264dc1a-907b-4104-9f42-e103b4d1519b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:30.431128 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c264dc1a-907b-4104-9f42-e103b4d1519b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:30.432479 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c264dc1a-907b-4104-9f42-e103b4d1519b None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:30.440422 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4af6dd23-4f2f-4eb9-abc2-759306c08bdb None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:30.441943 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4af6dd23-4f2f-4eb9-abc2-759306c08bdb None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:30.450507 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-c264dc1a-907b-4104-9f42-e103b4d1519b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:23:30.450507 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:30.450507 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:30.450507 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:30.450507 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:30.450507 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:30.450507 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:30.450507 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:30.450507 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:30.450507 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:30.450507 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:30.450507 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:30.450507 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 04:23:30.450507 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 04:23:30.450507 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 04:23:30.450507 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 04:23:30.451776 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:30.451776 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:30.451776 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:30.451776 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:30.451776 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:30.451776 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:30.451776 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:30.451776 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:30.451776 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:30.451776 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:30.451776 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:30.451776 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:30.451776 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:30.451776 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:30.451776 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:30.451776 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:30.451776 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 04:23:30.452911 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 04:23:30.452911 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:23:30.452911 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:23:30.454335 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 429/845] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:23:30 2022] GET /identity/v3/domains/Default => generated 87 bytes in 232 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:30.460026 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c264dc1a-907b-4104-9f42-e103b4d1519b None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:30.463214 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:30.584778 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 417/846] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Sep 25 04:23:30 2022] GET /identity/v3/auth/tokens => generated 1161 bytes in 407 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 25 04:23:30.590084 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4af6dd23-4f2f-4eb9-abc2-759306c08bdb None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:30.596384 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c264dc1a-907b-4104-9f42-e103b4d1519b None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:30.636385 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81db666d-d5c5-441f-9cd9-c897cebcfc9d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:30.637375 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81db666d-d5c5-441f-9cd9-c897cebcfc9d None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:30.638435 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81db666d-d5c5-441f-9cd9-c897cebcfc9d None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:30.638924 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81db666d-d5c5-441f-9cd9-c897cebcfc9d None admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:30.640830 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81db666d-d5c5-441f-9cd9-c897cebcfc9d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:30.641352 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81db666d-d5c5-441f-9cd9-c897cebcfc9d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:30.643644 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81db666d-d5c5-441f-9cd9-c897cebcfc9d None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:30.662705 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 430/847] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Sep 25 04:23:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:30.663485 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81db666d-d5c5-441f-9cd9-c897cebcfc9d None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:30.697187 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:30.715847 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4af6dd23-4f2f-4eb9-abc2-759306c08bdb None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:30.737266 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:30.760226 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ed463dd-828d-468d-85a6-c935f9e74b2e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:30.761402 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ed463dd-828d-468d-85a6-c935f9e74b2e None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:30.761975 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ed463dd-828d-468d-85a6-c935f9e74b2e None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:30.762448 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ed463dd-828d-468d-85a6-c935f9e74b2e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:30.774389 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81db666d-d5c5-441f-9cd9-c897cebcfc9d None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:30.775616 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ed463dd-828d-468d-85a6-c935f9e74b2e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:30.777483 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ed463dd-828d-468d-85a6-c935f9e74b2e None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:30.794065 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-7ed463dd-828d-468d-85a6-c935f9e74b2e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:23:30.794065 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:30.794065 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:30.794065 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:30.794065 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:30.794065 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:30.794065 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:30.794065 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:30.794065 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:30.794065 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:30.794065 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:30.794065 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:30.794065 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 04:23:30.794065 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 04:23:30.794065 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 04:23:30.794065 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 04:23:30.795094 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:30.795094 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:30.795094 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:30.795094 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:30.795094 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:30.795094 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:30.795094 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:30.795094 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:30.795094 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:30.795094 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:30.795094 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:30.795094 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:30.795094 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:30.795094 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:30.795094 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:30.795094 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:30.795094 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 04:23:30.796570 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 04:23:30.796570 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:23:30.796570 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:30.798382 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 418/848] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:23:30 2022] GET /identity/v3/domains/Default => generated 87 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:30.804215 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ed463dd-828d-468d-85a6-c935f9e74b2e None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:30.817836 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e24e1cc6-62d5-4eb5-9edd-a732a0866c46 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:30.818656 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e24e1cc6-62d5-4eb5-9edd-a732a0866c46 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:30.819411 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e24e1cc6-62d5-4eb5-9edd-a732a0866c46 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:30.819524 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e24e1cc6-62d5-4eb5-9edd-a732a0866c46 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:30.829068 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e24e1cc6-62d5-4eb5-9edd-a732a0866c46 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:30.830690 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e24e1cc6-62d5-4eb5-9edd-a732a0866c46 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:30.843711 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-e24e1cc6-62d5-4eb5-9edd-a732a0866c46 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 25 04:23:30.843711 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:30.843711 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:30.843711 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:30.843711 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:30.843711 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:30.843711 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:30.843711 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:30.843711 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:30.843711 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:30.843711 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:30.843711 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:30.843711 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 25 04:23:30.843711 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 25 04:23:30.843711 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 25 04:23:30.843711 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 25 04:23:30.844747 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:30.844747 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:30.844747 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 25 04:23:30.844747 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 04:23:30.844747 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 25 04:23:30.844747 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 04:23:30.844747 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:30.844747 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:30.844747 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:30.844747 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:30.844747 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:30.844747 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:30.844747 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:30.844747 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:30.844747 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:30.844747 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:30.844747 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:30.844747 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:30.845827 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:30.845827 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:30.845827 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 25 04:23:30.845827 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 25 04:23:30.845827 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 25 04:23:30.845827 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 25 04:23:30.845827 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 25 04:23:30.845827 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 25 04:23:30.845827 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 25 04:23:30.845827 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:23:30.850081 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 431/849] 10.222.0.25 () {56 vars in 1117 bytes} [Sun Sep 25 04:23:30 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 04:23:30.851240 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e24e1cc6-62d5-4eb5-9edd-a732a0866c46 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:30.931147 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ed463dd-828d-468d-85a6-c935f9e74b2e None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:30.944675 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:30.954936 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e24e1cc6-62d5-4eb5-9edd-a732a0866c46 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:30.970413 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72ee4a9f-c837-4043-8916-82eb3f81fd05 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:30.971042 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ee4a9f-c837-4043-8916-82eb3f81fd05 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:30.971316 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ee4a9f-c837-4043-8916-82eb3f81fd05 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:30.971584 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ee4a9f-c837-4043-8916-82eb3f81fd05 None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:30.972259 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72ee4a9f-c837-4043-8916-82eb3f81fd05 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:30.972519 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72ee4a9f-c837-4043-8916-82eb3f81fd05 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:30.973918 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72ee4a9f-c837-4043-8916-82eb3f81fd05 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:30.991516 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 419/850] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Sep 25 04:23:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:30.991516 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5bbb7ab-8427-4fea-84a4-b83382850d88 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:30.991516 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5bbb7ab-8427-4fea-84a4-b83382850d88 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:30.991516 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5bbb7ab-8427-4fea-84a4-b83382850d88 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:30.991516 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 420/851] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:23:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:23:30.993445 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5bbb7ab-8427-4fea-84a4-b83382850d88 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:31.000177 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2341d2f-006f-4e13-8375-df6a6aadb77b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:31.000828 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2341d2f-006f-4e13-8375-df6a6aadb77b None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:31.001107 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2341d2f-006f-4e13-8375-df6a6aadb77b None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:31.001371 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2341d2f-006f-4e13-8375-df6a6aadb77b None admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:31.002102 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2341d2f-006f-4e13-8375-df6a6aadb77b None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:31.002410 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2341d2f-006f-4e13-8375-df6a6aadb77b None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:31.003849 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2341d2f-006f-4e13-8375-df6a6aadb77b None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:31.027238 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 432/852] 10.222.0.25 () {56 vars in 1149 bytes} [Sun Sep 25 04:23:30 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:31.027238 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e4b04b8-ed5d-49f0-92fd-24c2581b4eda None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:31.027238 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e4b04b8-ed5d-49f0-92fd-24c2581b4eda None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:31.027238 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e4b04b8-ed5d-49f0-92fd-24c2581b4eda None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:31.034271 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:31.043006 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-5e4b04b8-ed5d-49f0-92fd-24c2581b4eda None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:23:31.044206 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-5e4b04b8-ed5d-49f0-92fd-24c2581b4eda None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:23:31.119348 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 433/853] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:23:31 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 04:23:31.120562 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e4b04b8-ed5d-49f0-92fd-24c2581b4eda None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:31.125507 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5bbb7ab-8427-4fea-84a4-b83382850d88 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:31.167229 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc856d0b-867f-4b96-91a9-e52dfbd9c71b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:31.168176 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc856d0b-867f-4b96-91a9-e52dfbd9c71b None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:31.168688 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc856d0b-867f-4b96-91a9-e52dfbd9c71b None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:31.169108 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc856d0b-867f-4b96-91a9-e52dfbd9c71b None admin] PATH_INFO: `/v3/projects/service` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:31.177616 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc856d0b-867f-4b96-91a9-e52dfbd9c71b None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:31.179324 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc856d0b-867f-4b96-91a9-e52dfbd9c71b None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:31.193458 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-fc856d0b-867f-4b96-91a9-e52dfbd9c71b None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 04:23:31.193458 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:31.193458 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:31.193458 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:31.193458 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:31.193458 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:31.193458 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:31.193458 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:31.193458 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:31.193458 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:31.193458 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:31.193458 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:31.193458 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 25 04:23:31.193458 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 25 04:23:31.193458 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 25 04:23:31.194474 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 25 04:23:31.194474 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:31.194474 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:31.194474 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:31.194474 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:31.194474 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:31.194474 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:31.194474 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:31.194474 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:31.194474 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:31.194474 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:31.194474 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:31.194474 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:31.194474 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:31.194474 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:31.194474 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:31.194474 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:31.195406 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 25 04:23:31.195406 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 25 04:23:31.195406 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 25 04:23:31.195406 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 25 04:23:31.195406 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 25 04:23:31.195406 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 25 04:23:31.195406 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 04:23:31.195406 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:31.201302 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 421/854] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Sep 25 04:23:30 2022] GET /identity/v3/projects/service => generated 88 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:31.204310 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc856d0b-867f-4b96-91a9-e52dfbd9c71b None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:31.220852 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:31.246269 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e4b04b8-ed5d-49f0-92fd-24c2581b4eda None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:31.292492 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9867099d-ee73-46f3-a667-324bef7972db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:31.293425 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9867099d-ee73-46f3-a667-324bef7972db None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:31.293894 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9867099d-ee73-46f3-a667-324bef7972db None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:31.294311 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9867099d-ee73-46f3-a667-324bef7972db None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:31.319733 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:31.337295 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc856d0b-867f-4b96-91a9-e52dfbd9c71b None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:31.385591 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af5bb7ba-050b-4a2f-b6bf-4a7d30041e25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:31.386332 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af5bb7ba-050b-4a2f-b6bf-4a7d30041e25 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:31.386708 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af5bb7ba-050b-4a2f-b6bf-4a7d30041e25 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:31.387054 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af5bb7ba-050b-4a2f-b6bf-4a7d30041e25 None admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:31.387833 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af5bb7ba-050b-4a2f-b6bf-4a7d30041e25 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:31.388212 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af5bb7ba-050b-4a2f-b6bf-4a7d30041e25 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:31.390497 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af5bb7ba-050b-4a2f-b6bf-4a7d30041e25 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:31.394590 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9867099d-ee73-46f3-a667-324bef7972db None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:31.396269 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9867099d-ee73-46f3-a667-324bef7972db None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:31.408773 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 422/855] 10.222.0.25 () {56 vars in 1148 bytes} [Sun Sep 25 04:23:31 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:31.411998 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af5bb7ba-050b-4a2f-b6bf-4a7d30041e25 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:31.510189 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:31.540084 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 434/856] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Sep 25 04:23:31 2022] GET /identity/v3/auth/tokens => generated 1161 bytes in 419 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 25 04:23:31.540084 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9867099d-ee73-46f3-a667-324bef7972db None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:31.549862 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af5bb7ba-050b-4a2f-b6bf-4a7d30041e25 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:31.588717 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-479bf8db-adfa-49a7-ac72-06e416686469 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:31.589331 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-479bf8db-adfa-49a7-ac72-06e416686469 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:31.589609 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-479bf8db-adfa-49a7-ac72-06e416686469 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:31.589880 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-479bf8db-adfa-49a7-ac72-06e416686469 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:31.590604 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-479bf8db-adfa-49a7-ac72-06e416686469 None admin] RBAC: Adding query filter params (role.id=c51fe7c5ad804040879acfb54269455f, scope.project.id=056ca1b1a9974a7581550de0d9217823, user.id=23ea88f46e7740d295148a93ade29dd9) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:31.590912 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-479bf8db-adfa-49a7-ac72-06e416686469 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:31.592349 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-479bf8db-adfa-49a7-ac72-06e416686469 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:31.600691 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 423/857] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Sep 25 04:23:31 2022] GET /identity/v3/role_assignments?user.id=23ea88f46e7740d295148a93ade29dd9&scope.project.id=056ca1b1a9974a7581550de0d9217823&role.id=c51fe7c5ad804040879acfb54269455f => generated 608 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:31.600997 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-479bf8db-adfa-49a7-ac72-06e416686469 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:31.623776 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:31.681299 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9867099d-ee73-46f3-a667-324bef7972db None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:31.731665 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6d22514-d181-4af8-8c13-35accd49e6ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:31.732322 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6d22514-d181-4af8-8c13-35accd49e6ce None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:31.732694 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6d22514-d181-4af8-8c13-35accd49e6ce None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:31.732886 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6d22514-d181-4af8-8c13-35accd49e6ce None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:31.734121 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6d22514-d181-4af8-8c13-35accd49e6ce None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:31.736202 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6d22514-d181-4af8-8c13-35accd49e6ce None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:31.745940 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-479bf8db-adfa-49a7-ac72-06e416686469 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:31.751669 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-c6d22514-d181-4af8-8c13-35accd49e6ce None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 25 04:23:31.751669 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:31.751669 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 25 04:23:31.751669 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 25 04:23:31.751669 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 25 04:23:31.751669 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 25 04:23:31.751669 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 25 04:23:31.751669 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 04:23:31.751669 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 25 04:23:31.751669 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:23:31.751669 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 25 04:23:31.751669 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:23:31.751669 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:31.751669 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:31.751669 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:31.751669 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:31.751669 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:31.752799 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:31.752799 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:31.752799 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:31.752799 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:31.752799 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:31.752799 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:31.752799 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 25 04:23:31.752799 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 25 04:23:31.752799 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 25 04:23:31.752799 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 25 04:23:31.752799 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:31.752799 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:31.752799 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:31.752799 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:31.752799 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:31.752799 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:31.753709 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:31.753709 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:31.753709 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:31.753709 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:31.753709 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:31.753709 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:31.753709 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:31.753709 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:31.753709 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:31.753709 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:31.753709 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 25 04:23:31.753709 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 04:23:31.753709 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 25 04:23:31.753709 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:23:31.759807 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 435/858] 10.222.0.25 () {56 vars in 1114 bytes} [Sun Sep 25 04:23:31 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:31.761828 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6d22514-d181-4af8-8c13-35accd49e6ce None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:31.794420 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3c8aafd-6329-452b-b661-12ce30fb991d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:31.795532 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3c8aafd-6329-452b-b661-12ce30fb991d None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:31.796521 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3c8aafd-6329-452b-b661-12ce30fb991d None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:31.797163 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3c8aafd-6329-452b-b661-12ce30fb991d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:31.811898 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3c8aafd-6329-452b-b661-12ce30fb991d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:31.813510 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3c8aafd-6329-452b-b661-12ce30fb991d None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:31.829773 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:31.830036 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-c3c8aafd-6329-452b-b661-12ce30fb991d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:23:31.830036 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:31.830036 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:31.830036 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:31.830036 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:31.830036 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:31.830036 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:31.830036 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:31.830036 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:31.830036 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:31.830036 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:31.830036 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:31.830036 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 04:23:31.830036 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 04:23:31.830036 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 04:23:31.830036 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 04:23:31.831648 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:31.831648 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:31.831648 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:31.831648 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:31.831648 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:31.831648 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:31.831648 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:31.831648 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:31.831648 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:31.831648 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:31.831648 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:31.831648 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:31.831648 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:31.831648 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:31.831648 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:31.831648 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:31.831648 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 04:23:31.833339 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 04:23:31.833339 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:23:31.833339 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:31.833339 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 424/859] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:23:31 2022] GET /identity/v3/domains/Default => generated 87 bytes in 230 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:31.837695 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3c8aafd-6329-452b-b661-12ce30fb991d None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:31.914353 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6d22514-d181-4af8-8c13-35accd49e6ce None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:31.934018 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:31.962082 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54438f3b-823f-4af3-b695-44539d39a5f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:31.963158 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54438f3b-823f-4af3-b695-44539d39a5f5 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:31.963656 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54438f3b-823f-4af3-b695-44539d39a5f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:31.964056 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54438f3b-823f-4af3-b695-44539d39a5f5 None admin] PATH_INFO: `/v3/services` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:31.964841 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54438f3b-823f-4af3-b695-44539d39a5f5 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:31.965248 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54438f3b-823f-4af3-b695-44539d39a5f5 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:31.966831 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54438f3b-823f-4af3-b695-44539d39a5f5 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:31.973852 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 436/860] 10.222.0.25 () {56 vars in 1114 bytes} [Sun Sep 25 04:23:31 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:31.980454 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54438f3b-823f-4af3-b695-44539d39a5f5 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:31.995740 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3c8aafd-6329-452b-b661-12ce30fb991d None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:32.037290 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93ade1e1-20c3-40a2-bdcf-84e0257a0547 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:32.038306 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93ade1e1-20c3-40a2-bdcf-84e0257a0547 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:32.038798 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93ade1e1-20c3-40a2-bdcf-84e0257a0547 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:32.039294 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93ade1e1-20c3-40a2-bdcf-84e0257a0547 None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:32.040065 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93ade1e1-20c3-40a2-bdcf-84e0257a0547 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:32.040446 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93ade1e1-20c3-40a2-bdcf-84e0257a0547 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:32.041994 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93ade1e1-20c3-40a2-bdcf-84e0257a0547 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:32.060348 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 425/861] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Sep 25 04:23:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:32.061591 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93ade1e1-20c3-40a2-bdcf-84e0257a0547 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:32.134506 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54438f3b-823f-4af3-b695-44539d39a5f5 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:32.147890 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:32.183480 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-710f199b-4faa-4a1e-8e64-f0d241bf9334 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:32.184525 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-710f199b-4faa-4a1e-8e64-f0d241bf9334 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:32.184997 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-710f199b-4faa-4a1e-8e64-f0d241bf9334 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:32.185439 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-710f199b-4faa-4a1e-8e64-f0d241bf9334 None admin] PATH_INFO: `/v3/services` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:32.186233 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-710f199b-4faa-4a1e-8e64-f0d241bf9334 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:32.187320 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-710f199b-4faa-4a1e-8e64-f0d241bf9334 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:32.189323 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-710f199b-4faa-4a1e-8e64-f0d241bf9334 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:32.201307 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 437/862] 10.222.0.25 () {56 vars in 1114 bytes} [Sun Sep 25 04:23:31 2022] GET /identity/v3/services?type=volumev3 => generated 125 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:32.220287 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93ade1e1-20c3-40a2-bdcf-84e0257a0547 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:32.250663 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:32.263212 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f47f33fb-2602-4271-8053-defa7fa8de2c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:32.263863 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f47f33fb-2602-4271-8053-defa7fa8de2c None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:32.264142 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f47f33fb-2602-4271-8053-defa7fa8de2c None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:32.264416 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f47f33fb-2602-4271-8053-defa7fa8de2c None admin] PATH_INFO: `/v3/projects/service` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:32.273149 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f47f33fb-2602-4271-8053-defa7fa8de2c None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:32.274639 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f47f33fb-2602-4271-8053-defa7fa8de2c None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:32.405708 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-f47f33fb-2602-4271-8053-defa7fa8de2c None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 04:23:32.405708 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:32.405708 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:32.405708 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:32.405708 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:32.405708 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:32.405708 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:32.405708 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:32.405708 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:32.405708 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:32.405708 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:32.405708 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:32.405708 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 25 04:23:32.405708 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 25 04:23:32.405708 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 25 04:23:32.406690 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 25 04:23:32.406690 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:32.406690 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:32.406690 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:32.406690 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:32.406690 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:32.406690 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:32.406690 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:32.406690 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:32.406690 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:32.406690 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:32.406690 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:32.406690 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:32.406690 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:32.406690 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:32.406690 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:32.406690 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:32.407648 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 25 04:23:32.407648 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 25 04:23:32.407648 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 25 04:23:32.407648 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 25 04:23:32.407648 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 25 04:23:32.407648 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 25 04:23:32.407648 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 04:23:32.407648 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:32.407648 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 426/863] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Sep 25 04:23:32 2022] GET /identity/v3/projects/service => generated 88 bytes in 346 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 04:23:32.414396 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-710f199b-4faa-4a1e-8e64-f0d241bf9334 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:32.524631 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-710f199b-4faa-4a1e-8e64-f0d241bf9334 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:32.564522 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43ac2944-0155-4cdb-b580-4187019bd48b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:32.565179 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43ac2944-0155-4cdb-b580-4187019bd48b None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:32.565457 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43ac2944-0155-4cdb-b580-4187019bd48b None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:32.565724 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43ac2944-0155-4cdb-b580-4187019bd48b None admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:32.566419 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43ac2944-0155-4cdb-b580-4187019bd48b None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:32.566723 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43ac2944-0155-4cdb-b580-4187019bd48b None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:32.568150 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43ac2944-0155-4cdb-b580-4187019bd48b None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:32.581560 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 438/864] 10.222.0.25 () {56 vars in 1148 bytes} [Sun Sep 25 04:23:32 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:32.587445 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f47f33fb-2602-4271-8053-defa7fa8de2c None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:32.653272 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f109658-7e93-4d7b-8d5c-aa80cc2daa90 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:32.653561 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f109658-7e93-4d7b-8d5c-aa80cc2daa90 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:32.653856 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f109658-7e93-4d7b-8d5c-aa80cc2daa90 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:32.655964 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 439/865] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:23:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 25 04:23:32.661573 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76004497-9e3d-49e8-ade5-1961da4f4881 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:32.661858 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76004497-9e3d-49e8-ade5-1961da4f4881 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:32.662137 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76004497-9e3d-49e8-ade5-1961da4f4881 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:32.682915 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-76004497-9e3d-49e8-ade5-1961da4f4881 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:23:32.684286 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-76004497-9e3d-49e8-ade5-1961da4f4881 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:23:32.707319 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f47f33fb-2602-4271-8053-defa7fa8de2c None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:32.724669 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:32.760621 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ec5123e-25ce-400c-8520-b606495d1ca5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:32.761798 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ec5123e-25ce-400c-8520-b606495d1ca5 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:32.762523 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ec5123e-25ce-400c-8520-b606495d1ca5 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:32.763353 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ec5123e-25ce-400c-8520-b606495d1ca5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:32.764544 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ec5123e-25ce-400c-8520-b606495d1ca5 None admin] RBAC: Adding query filter params (role.id=9b44520f482c4423bd10568d0f86718c, scope.project.id=056ca1b1a9974a7581550de0d9217823, user.id=050b3aab44bb4b17aa4a1ba7cccdb231) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:32.765087 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ec5123e-25ce-400c-8520-b606495d1ca5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:32.766780 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ec5123e-25ce-400c-8520-b606495d1ca5 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:32.775851 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 440/866] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:23:32 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 25 04:23:32.777525 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 427/867] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Sep 25 04:23:32 2022] GET /identity/v3/role_assignments?user.id=050b3aab44bb4b17aa4a1ba7cccdb231&scope.project.id=056ca1b1a9974a7581550de0d9217823&role.id=9b44520f482c4423bd10568d0f86718c => generated 608 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:33.064964 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76004497-9e3d-49e8-ade5-1961da4f4881 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:33.172233 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76004497-9e3d-49e8-ade5-1961da4f4881 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:33.208763 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-392ab680-c59b-47cb-8486-abe9a0ad825c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:33.209436 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-392ab680-c59b-47cb-8486-abe9a0ad825c None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:33.209795 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-392ab680-c59b-47cb-8486-abe9a0ad825c None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:33.210078 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-392ab680-c59b-47cb-8486-abe9a0ad825c None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:33.210680 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-392ab680-c59b-47cb-8486-abe9a0ad825c None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:33.212098 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-392ab680-c59b-47cb-8486-abe9a0ad825c None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:33.222806 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-392ab680-c59b-47cb-8486-abe9a0ad825c None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 25 04:23:33.222806 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:33.222806 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 25 04:23:33.222806 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 25 04:23:33.222806 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 25 04:23:33.222806 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 25 04:23:33.222806 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 25 04:23:33.222806 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 04:23:33.222806 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 25 04:23:33.222806 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:23:33.222806 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 25 04:23:33.222806 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:23:33.222806 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:33.222806 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:33.222806 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:33.222806 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:33.223904 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:33.223904 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:33.223904 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:33.223904 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:33.223904 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:33.223904 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:33.223904 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:33.223904 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 25 04:23:33.223904 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 25 04:23:33.223904 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 25 04:23:33.223904 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 25 04:23:33.223904 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:33.223904 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:33.223904 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:33.223904 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:33.223904 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:33.223904 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:33.225539 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:33.225539 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:33.225539 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:33.225539 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:33.225539 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:33.225539 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:33.225539 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:33.225539 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:33.225539 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:33.225539 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:33.225539 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 25 04:23:33.225539 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 04:23:33.225539 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 25 04:23:33.225539 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:23:33.225539 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 441/868] 10.222.0.25 () {56 vars in 1132 bytes} [Sun Sep 25 04:23:33 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:33.230143 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ec5123e-25ce-400c-8520-b606495d1ca5 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:33.297402 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:33.339889 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ec5123e-25ce-400c-8520-b606495d1ca5 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:33.375394 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d86065f-ed2d-448d-baad-a5acdbea8ed6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:33.376055 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d86065f-ed2d-448d-baad-a5acdbea8ed6 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:33.376342 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d86065f-ed2d-448d-baad-a5acdbea8ed6 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:33.376685 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d86065f-ed2d-448d-baad-a5acdbea8ed6 None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:33.377320 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d86065f-ed2d-448d-baad-a5acdbea8ed6 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:33.377626 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d86065f-ed2d-448d-baad-a5acdbea8ed6 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:33.379001 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d86065f-ed2d-448d-baad-a5acdbea8ed6 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:33.384149 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 428/869] 10.222.0.25 () {56 vars in 1126 bytes} [Sun Sep 25 04:23:33 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:33.389984 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-392ab680-c59b-47cb-8486-abe9a0ad825c None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:33.419707 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:33.498381 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-392ab680-c59b-47cb-8486-abe9a0ad825c None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:33.534740 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed519a50-232c-4d53-a5fc-66d37461070f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:33.535688 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed519a50-232c-4d53-a5fc-66d37461070f None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:33.536152 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed519a50-232c-4d53-a5fc-66d37461070f None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:33.536607 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed519a50-232c-4d53-a5fc-66d37461070f None admin] PATH_INFO: `/v3/services` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:33.537444 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed519a50-232c-4d53-a5fc-66d37461070f None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:33.537929 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed519a50-232c-4d53-a5fc-66d37461070f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:33.539464 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed519a50-232c-4d53-a5fc-66d37461070f None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:33.544610 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 442/870] 10.222.0.25 () {56 vars in 1126 bytes} [Sun Sep 25 04:23:33 2022] GET /identity/v3/services?type=compute_legacy => generated 131 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:23:34.003666 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07bbb307-a980-44d5-8a81-e412b7a9ad64 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:34.004220 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07bbb307-a980-44d5-8a81-e412b7a9ad64 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:34.004220 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07bbb307-a980-44d5-8a81-e412b7a9ad64 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:34.005795 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 429/871] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:23:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:23:34.011337 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6222d2cb-495c-40c0-b354-3d38d1c5b022 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:34.011628 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6222d2cb-495c-40c0-b354-3d38d1c5b022 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:34.011904 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6222d2cb-495c-40c0-b354-3d38d1c5b022 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:34.030759 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-6222d2cb-495c-40c0-b354-3d38d1c5b022 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:23:34.032965 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-6222d2cb-495c-40c0-b354-3d38d1c5b022 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:23:34.111179 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 443/872] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:23:34 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 04:23:34.385473 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07bbb307-a980-44d5-8a81-e412b7a9ad64 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:34.509570 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07bbb307-a980-44d5-8a81-e412b7a9ad64 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:34.525036 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:34.546181 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22018699-5eda-4d11-b277-c3558482b413 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:34.546883 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22018699-5eda-4d11-b277-c3558482b413 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:34.547165 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22018699-5eda-4d11-b277-c3558482b413 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:34.547438 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22018699-5eda-4d11-b277-c3558482b413 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:34.548021 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22018699-5eda-4d11-b277-c3558482b413 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:34.549413 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22018699-5eda-4d11-b277-c3558482b413 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:34.558838 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-22018699-5eda-4d11-b277-c3558482b413 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 25 04:23:34.558838 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:34.558838 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 25 04:23:34.558838 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 25 04:23:34.558838 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 25 04:23:34.558838 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 25 04:23:34.558838 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 25 04:23:34.558838 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 04:23:34.558838 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 25 04:23:34.558838 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:34.558838 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 25 04:23:34.558838 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:34.558838 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:34.558838 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:34.558838 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:34.558838 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:34.558838 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:34.559883 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:34.559883 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:34.559883 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:34.559883 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:34.559883 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:34.559883 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:34.559883 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 25 04:23:34.559883 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 25 04:23:34.559883 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 25 04:23:34.559883 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 25 04:23:34.559883 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:34.559883 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:34.559883 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:34.559883 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:34.559883 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:34.559883 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:34.561160 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:34.561160 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:34.561160 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:34.561160 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:34.561160 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:34.561160 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:34.561160 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:34.561160 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:34.561160 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:34.561160 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:34.561160 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 25 04:23:34.561160 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 04:23:34.561160 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 25 04:23:34.561160 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:34.561160 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 430/873] 10.222.0.25 () {56 vars in 1126 bytes} [Sun Sep 25 04:23:34 2022] GET /identity/v3/services/object-store => generated 93 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:34.569714 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6222d2cb-495c-40c0-b354-3d38d1c5b022 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:34.590374 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-993a3a8b-041a-4a14-9340-1bdaf8b51ee1 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:34.590644 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-993a3a8b-041a-4a14-9340-1bdaf8b51ee1 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:34.591069 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-993a3a8b-041a-4a14-9340-1bdaf8b51ee1 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:34.591827 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:34.594284 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 431/874] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:23:34 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:23:34.598520 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36de49dc-7c35-44df-a108-b63f9713821f None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:34.598803 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36de49dc-7c35-44df-a108-b63f9713821f None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:34.599076 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36de49dc-7c35-44df-a108-b63f9713821f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:34.618031 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-36de49dc-7c35-44df-a108-b63f9713821f None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:23:34.619291 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-36de49dc-7c35-44df-a108-b63f9713821f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:23:34.678514 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6222d2cb-495c-40c0-b354-3d38d1c5b022 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:34.690828 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 432/875] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:23:34 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 04:23:34.716241 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edae3c74-16d5-4f80-8ca5-f718904fdb39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:34.716948 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edae3c74-16d5-4f80-8ca5-f718904fdb39 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:34.717235 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edae3c74-16d5-4f80-8ca5-f718904fdb39 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:34.717506 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edae3c74-16d5-4f80-8ca5-f718904fdb39 None admin] PATH_INFO: `/v3/services` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:34.718758 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edae3c74-16d5-4f80-8ca5-f718904fdb39 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:34.719059 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edae3c74-16d5-4f80-8ca5-f718904fdb39 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:34.720487 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edae3c74-16d5-4f80-8ca5-f718904fdb39 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:34.725530 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 444/876] 10.222.0.25 () {56 vars in 1122 bytes} [Sun Sep 25 04:23:34 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:34.734604 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36de49dc-7c35-44df-a108-b63f9713821f None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:34.839718 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36de49dc-7c35-44df-a108-b63f9713821f None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:34.848718 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:34.876967 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89e43b8e-bbe7-4153-9be6-f220e51d6ab9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:34.877580 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89e43b8e-bbe7-4153-9be6-f220e51d6ab9 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:34.877860 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89e43b8e-bbe7-4153-9be6-f220e51d6ab9 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:34.878159 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89e43b8e-bbe7-4153-9be6-f220e51d6ab9 None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:34.878766 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89e43b8e-bbe7-4153-9be6-f220e51d6ab9 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:34.879071 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89e43b8e-bbe7-4153-9be6-f220e51d6ab9 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:34.880445 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89e43b8e-bbe7-4153-9be6-f220e51d6ab9 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:34.885364 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 433/877] 10.222.0.25 () {56 vars in 1122 bytes} [Sun Sep 25 04:23:34 2022] GET /identity/v3/services?type=object-store => generated 129 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:34.973736 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edae3c74-16d5-4f80-8ca5-f718904fdb39 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:35.072696 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:35.091250 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edae3c74-16d5-4f80-8ca5-f718904fdb39 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:35.106051 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8a1086b-96d0-4519-9592-9f2a60aa19ee None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:35.106585 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8a1086b-96d0-4519-9592-9f2a60aa19ee None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:35.106585 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8a1086b-96d0-4519-9592-9f2a60aa19ee None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:35.108823 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 434/878] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:23:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 25 04:23:35.113859 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fc078b4-c11c-4134-8df6-b07ca653f2c6 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:35.114292 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fc078b4-c11c-4134-8df6-b07ca653f2c6 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:35.114572 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fc078b4-c11c-4134-8df6-b07ca653f2c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:35.129531 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-689fae4d-e15b-4a2d-a5d3-20fd76ff476e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:35.130492 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-689fae4d-e15b-4a2d-a5d3-20fd76ff476e None admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:35.130988 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-689fae4d-e15b-4a2d-a5d3-20fd76ff476e None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:35.131460 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-689fae4d-e15b-4a2d-a5d3-20fd76ff476e None admin] PATH_INFO: `/v3/services` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:35.132343 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-689fae4d-e15b-4a2d-a5d3-20fd76ff476e None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:35.132992 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-0fc078b4-c11c-4134-8df6-b07ca653f2c6 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:23:35.134174 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-0fc078b4-c11c-4134-8df6-b07ca653f2c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:23:35.134382 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-689fae4d-e15b-4a2d-a5d3-20fd76ff476e None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:35.152242 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 445/879] 10.222.0.25 () {60 vars in 1141 bytes} [Sun Sep 25 04:23:34 2022] POST /identity/v3/services => generated 251 bytes in 179 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:23:35.219257 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 435/880] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:23:35 2022] POST /identity/v3/auth/tokens => generated 1262 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 04:23:35.502486 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-689fae4d-e15b-4a2d-a5d3-20fd76ff476e None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:35.607310 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-689fae4d-e15b-4a2d-a5d3-20fd76ff476e None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:35.644148 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-303806c2-7d27-4d34-9cd5-98dd45188407 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:35.644819 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-303806c2-7d27-4d34-9cd5-98dd45188407 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:35.645101 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-303806c2-7d27-4d34-9cd5-98dd45188407 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:35.645367 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-303806c2-7d27-4d34-9cd5-98dd45188407 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:35.657095 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-303806c2-7d27-4d34-9cd5-98dd45188407 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 04:23:35.662399 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-303806c2-7d27-4d34-9cd5-98dd45188407 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 25 04:23:35.662399 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:35.662399 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:35.662399 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:35.662399 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:35.662399 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:35.662399 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:35.662399 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:35.662399 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:35.662399 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:35.662399 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:35.662399 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:35.662399 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 04:23:35.662399 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 04:23:35.662399 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 04:23:35.662399 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 04:23:35.663380 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 04:23:35.663380 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 04:23:35.663380 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 04:23:35.663380 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 04:23:35.663380 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:35.663380 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:35.663380 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:35.663380 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:35.663380 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:35.663380 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:35.663380 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:35.663380 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:35.663380 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:35.663380 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:35.663380 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:35.663380 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:35.664290 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:35.664290 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:35.664290 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:35.664290 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:35.664290 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 04:23:35.664290 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 04:23:35.664290 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 04:23:35.664290 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 04:23:35.664290 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 04:23:35.664290 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 04:23:35.664290 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 25 04:23:35.664290 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:23:35.664290 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 446/881] 10.222.0.25 () {56 vars in 1120 bytes} [Sun Sep 25 04:23:35 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:35.669767 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fc078b4-c11c-4134-8df6-b07ca653f2c6 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:35.769680 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:35.776006 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fc078b4-c11c-4134-8df6-b07ca653f2c6 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:35.812175 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4b2f1ee-fc9f-48f3-92a5-9e4a3f6d04ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:35.812813 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4b2f1ee-fc9f-48f3-92a5-9e4a3f6d04ba None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:35.813093 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4b2f1ee-fc9f-48f3-92a5-9e4a3f6d04ba None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:35.813365 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4b2f1ee-fc9f-48f3-92a5-9e4a3f6d04ba None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:35.814028 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4b2f1ee-fc9f-48f3-92a5-9e4a3f6d04ba None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:35.814332 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4b2f1ee-fc9f-48f3-92a5-9e4a3f6d04ba None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:35.815857 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4b2f1ee-fc9f-48f3-92a5-9e4a3f6d04ba None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:35.825302 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 436/882] 10.222.0.25 () {56 vars in 1115 bytes} [Sun Sep 25 04:23:35 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:35.832103 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-303806c2-7d27-4d34-9cd5-98dd45188407 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:35.833725 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ad4c9fe-f0a5-46f4-a043-b4706af0ade0 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:35.834015 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ad4c9fe-f0a5-46f4-a043-b4706af0ade0 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:35.834291 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ad4c9fe-f0a5-46f4-a043-b4706af0ade0 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:35.835925 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 437/883] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:23:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:23:35.836129 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:35.841755 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96cc1228-fb7d-4e1a-b515-fcee6cb8fd52 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:35.842076 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96cc1228-fb7d-4e1a-b515-fcee6cb8fd52 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:35.842303 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96cc1228-fb7d-4e1a-b515-fcee6cb8fd52 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:35.861294 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-96cc1228-fb7d-4e1a-b515-fcee6cb8fd52 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:23:35.862502 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-96cc1228-fb7d-4e1a-b515-fcee6cb8fd52 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:23:35.940009 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 438/884] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:23:35 2022] POST /identity/v3/auth/tokens => generated 1262 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 25 04:23:35.957169 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-303806c2-7d27-4d34-9cd5-98dd45188407 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:35.991738 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26e6b65e-7640-45e5-a2d5-5a5998b6f51f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:35.992590 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e6b65e-7640-45e5-a2d5-5a5998b6f51f None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:35.993030 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e6b65e-7640-45e5-a2d5-5a5998b6f51f None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:35.993465 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e6b65e-7640-45e5-a2d5-5a5998b6f51f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:36.004510 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26e6b65e-7640-45e5-a2d5-5a5998b6f51f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:36.006144 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26e6b65e-7640-45e5-a2d5-5a5998b6f51f None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:36.021260 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-26e6b65e-7640-45e5-a2d5-5a5998b6f51f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:23:36.021260 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:36.021260 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:36.021260 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:36.021260 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:36.021260 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:36.021260 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:36.021260 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:36.021260 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:36.021260 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:36.021260 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:36.021260 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:36.021260 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 04:23:36.021260 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 04:23:36.021260 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 04:23:36.021260 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 04:23:36.022455 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:36.022455 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:36.022455 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:36.022455 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:36.022455 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:36.022455 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:36.022455 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:36.022455 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:36.022455 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:36.022455 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:36.022455 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:36.022455 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:36.022455 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:36.022455 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:36.022455 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:36.022455 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:36.022455 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 04:23:36.023592 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 04:23:36.023592 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:23:36.023592 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:23:36.025248 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 447/885] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:23:35 2022] GET /identity/v3/domains/Default => generated 87 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:36.031130 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96cc1228-fb7d-4e1a-b515-fcee6cb8fd52 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:36.128424 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:36.142418 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96cc1228-fb7d-4e1a-b515-fcee6cb8fd52 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:36.177297 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-136b08db-e5f4-4188-a5e5-93b8b4d2abc9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:36.178103 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-136b08db-e5f4-4188-a5e5-93b8b4d2abc9 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:36.178559 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-136b08db-e5f4-4188-a5e5-93b8b4d2abc9 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:36.178980 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-136b08db-e5f4-4188-a5e5-93b8b4d2abc9 None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:36.179757 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-136b08db-e5f4-4188-a5e5-93b8b4d2abc9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:36.180202 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-136b08db-e5f4-4188-a5e5-93b8b4d2abc9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:36.181723 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-136b08db-e5f4-4188-a5e5-93b8b4d2abc9 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:36.193389 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 439/886] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Sep 25 04:23:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:36.198996 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26e6b65e-7640-45e5-a2d5-5a5998b6f51f None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:36.226178 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-136b08db-e5f4-4188-a5e5-93b8b4d2abc9 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:36.264407 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:36.317631 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26e6b65e-7640-45e5-a2d5-5a5998b6f51f None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:36.354686 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-136b08db-e5f4-4188-a5e5-93b8b4d2abc9 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:36.357123 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bec9f37a-ab82-4c68-a22d-92a20f53f6ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:36.357794 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bec9f37a-ab82-4c68-a22d-92a20f53f6ba None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:36.359282 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bec9f37a-ab82-4c68-a22d-92a20f53f6ba None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:36.359578 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bec9f37a-ab82-4c68-a22d-92a20f53f6ba None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:36.368911 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bec9f37a-ab82-4c68-a22d-92a20f53f6ba None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:36.370623 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bec9f37a-ab82-4c68-a22d-92a20f53f6ba None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:36.384552 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-bec9f37a-ab82-4c68-a22d-92a20f53f6ba None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 25 04:23:36.384552 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:36.384552 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:36.384552 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:36.384552 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:36.384552 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:36.384552 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:36.384552 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:36.384552 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:36.384552 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:36.384552 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:36.384552 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:36.384552 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 25 04:23:36.384552 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 25 04:23:36.384552 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 25 04:23:36.384552 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 25 04:23:36.385576 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:36.385576 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:36.385576 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 25 04:23:36.385576 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 04:23:36.385576 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 25 04:23:36.385576 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 04:23:36.385576 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:36.385576 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:36.385576 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:36.385576 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:36.385576 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:36.385576 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:36.385576 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:36.385576 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:36.385576 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:36.385576 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:36.385576 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:36.385576 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:36.386554 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:36.386554 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:36.386554 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 25 04:23:36.386554 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 25 04:23:36.386554 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 25 04:23:36.386554 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 25 04:23:36.386554 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 25 04:23:36.386554 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 25 04:23:36.386554 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 25 04:23:36.386554 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:23:36.388710 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 448/887] 10.222.0.25 () {56 vars in 1117 bytes} [Sun Sep 25 04:23:36 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:36.391512 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bec9f37a-ab82-4c68-a22d-92a20f53f6ba None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:36.403702 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-355d2545-198f-4acf-9cfb-43021183a63a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:36.404858 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-355d2545-198f-4acf-9cfb-43021183a63a None admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:36.406023 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-355d2545-198f-4acf-9cfb-43021183a63a None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:36.406504 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-355d2545-198f-4acf-9cfb-43021183a63a None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:36.407440 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-355d2545-198f-4acf-9cfb-43021183a63a None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:36.409780 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-355d2545-198f-4acf-9cfb-43021183a63a None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:36.432247 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 440/888] 10.222.0.25 () {60 vars in 1141 bytes} [Sun Sep 25 04:23:36 2022] POST /identity/v3/services => generated 269 bytes in 206 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 04:23:36.517274 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bec9f37a-ab82-4c68-a22d-92a20f53f6ba None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:36.536503 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:36.563357 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5552fa9-96d4-4cd1-8eaf-e138a3e9f041 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:36.563978 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5552fa9-96d4-4cd1-8eaf-e138a3e9f041 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:36.564252 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5552fa9-96d4-4cd1-8eaf-e138a3e9f041 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:36.564521 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5552fa9-96d4-4cd1-8eaf-e138a3e9f041 None admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:36.565292 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5552fa9-96d4-4cd1-8eaf-e138a3e9f041 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:36.565596 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5552fa9-96d4-4cd1-8eaf-e138a3e9f041 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:36.567120 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5552fa9-96d4-4cd1-8eaf-e138a3e9f041 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:36.588708 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 449/889] 10.222.0.25 () {56 vars in 1149 bytes} [Sun Sep 25 04:23:36 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:23:36.598212 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-355d2545-198f-4acf-9cfb-43021183a63a None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:36.611272 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:36.708895 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-355d2545-198f-4acf-9cfb-43021183a63a None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:36.744003 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01f20c7f-6177-433d-9cbd-76bce90e170a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:36.744625 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f20c7f-6177-433d-9cbd-76bce90e170a None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:36.744929 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f20c7f-6177-433d-9cbd-76bce90e170a None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:36.745296 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f20c7f-6177-433d-9cbd-76bce90e170a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:36.816672 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01f20c7f-6177-433d-9cbd-76bce90e170a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:36.818241 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01f20c7f-6177-433d-9cbd-76bce90e170a None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:36.829886 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:36.926676 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 441/890] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Sep 25 04:23:36 2022] GET /identity/v3/auth/tokens => generated 1372 bytes in 329 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 25 04:23:36.936980 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5552fa9-96d4-4cd1-8eaf-e138a3e9f041 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:37.040298 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5552fa9-96d4-4cd1-8eaf-e138a3e9f041 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:37.075019 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06b472ad-6bb7-40d3-b85f-8e11df3678b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:37.075650 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b472ad-6bb7-40d3-b85f-8e11df3678b1 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:37.076178 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b472ad-6bb7-40d3-b85f-8e11df3678b1 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:37.076446 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b472ad-6bb7-40d3-b85f-8e11df3678b1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:37.086628 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06b472ad-6bb7-40d3-b85f-8e11df3678b1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:37.088167 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06b472ad-6bb7-40d3-b85f-8e11df3678b1 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:37.104284 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-06b472ad-6bb7-40d3-b85f-8e11df3678b1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:23:37.104284 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:37.104284 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:37.104284 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:37.104284 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:37.104284 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:37.104284 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:37.104284 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:37.104284 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:37.104284 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:37.104284 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:37.104284 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:37.104284 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 04:23:37.104284 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 04:23:37.104284 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 04:23:37.104284 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 04:23:37.105340 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:37.105340 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:37.105340 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:37.105340 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:37.105340 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:37.105340 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:37.105340 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:37.105340 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:37.105340 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:37.105340 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:37.105340 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:37.105340 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:37.105340 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:37.105340 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:37.105340 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:37.105340 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:37.105340 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 04:23:37.106292 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 04:23:37.106292 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:23:37.106292 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:23:37.106292 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 450/891] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:23:36 2022] GET /identity/v3/domains/Default => generated 87 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:37.111519 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01f20c7f-6177-433d-9cbd-76bce90e170a None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:37.222280 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01f20c7f-6177-433d-9cbd-76bce90e170a None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:37.231805 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:37.258463 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fcadc22-3752-45da-803d-8ce9705f5b6b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:37.259079 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fcadc22-3752-45da-803d-8ce9705f5b6b None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:37.259363 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fcadc22-3752-45da-803d-8ce9705f5b6b None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:37.259634 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fcadc22-3752-45da-803d-8ce9705f5b6b None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:37.260347 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fcadc22-3752-45da-803d-8ce9705f5b6b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:37.260665 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fcadc22-3752-45da-803d-8ce9705f5b6b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:37.262058 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fcadc22-3752-45da-803d-8ce9705f5b6b None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:37.275365 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 442/892] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Sep 25 04:23:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:37.280727 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-877c9f54-777c-4e29-94f5-9724a07978b3 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:37.281042 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-877c9f54-777c-4e29-94f5-9724a07978b3 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:37.281280 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-877c9f54-777c-4e29-94f5-9724a07978b3 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:37.281406 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fcadc22-3752-45da-803d-8ce9705f5b6b None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:37.282953 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 451/893] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:23:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:23:37.288765 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ff9cbb7-bf76-4811-a93f-622ff85a7202 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:37.289108 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ff9cbb7-bf76-4811-a93f-622ff85a7202 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:37.289311 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ff9cbb7-bf76-4811-a93f-622ff85a7202 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:37.310002 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-2ff9cbb7-bf76-4811-a93f-622ff85a7202 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:23:37.312105 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-2ff9cbb7-bf76-4811-a93f-622ff85a7202 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:23:37.396699 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 452/894] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:23:37 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 25 04:23:37.401029 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:37.403864 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1835a65b-1056-4674-a517-40fb2630443d None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:37.405098 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1835a65b-1056-4674-a517-40fb2630443d None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:37.405622 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1835a65b-1056-4674-a517-40fb2630443d None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:37.407692 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 453/895] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:23:37 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:23:37.414081 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0ac9426-5d6d-4ed1-b424-b200d5b4c94b None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:37.414630 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0ac9426-5d6d-4ed1-b424-b200d5b4c94b None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:37.415113 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0ac9426-5d6d-4ed1-b424-b200d5b4c94b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:37.420089 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fcadc22-3752-45da-803d-8ce9705f5b6b None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:37.435496 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-c0ac9426-5d6d-4ed1-b424-b200d5b4c94b None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:23:37.436957 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-c0ac9426-5d6d-4ed1-b424-b200d5b4c94b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:23:37.458002 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c231b2aa-067a-4dda-b41d-294d7a2d291d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:37.458962 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c231b2aa-067a-4dda-b41d-294d7a2d291d None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:37.459477 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c231b2aa-067a-4dda-b41d-294d7a2d291d None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:37.459957 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c231b2aa-067a-4dda-b41d-294d7a2d291d None admin] PATH_INFO: `/v3/projects/service` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:37.474148 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c231b2aa-067a-4dda-b41d-294d7a2d291d None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:37.481740 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c231b2aa-067a-4dda-b41d-294d7a2d291d None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:37.498040 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-c231b2aa-067a-4dda-b41d-294d7a2d291d None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 04:23:37.498040 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:37.498040 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:37.498040 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:37.498040 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:37.498040 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:37.498040 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:37.498040 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:37.498040 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:37.498040 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:37.498040 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:37.498040 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:37.498040 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 25 04:23:37.498040 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 25 04:23:37.498040 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 25 04:23:37.499624 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 25 04:23:37.499624 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:37.499624 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:37.499624 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:37.499624 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:37.499624 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:37.499624 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:37.499624 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:37.499624 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:37.499624 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:37.499624 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:37.499624 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:37.499624 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:37.499624 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:37.499624 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:37.499624 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:37.499624 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:37.500551 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 25 04:23:37.500551 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 25 04:23:37.500551 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 25 04:23:37.500551 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 25 04:23:37.500551 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 25 04:23:37.500551 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 25 04:23:37.500551 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 04:23:37.500551 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:37.503549 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 443/896] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Sep 25 04:23:37 2022] GET /identity/v3/projects/service => generated 88 bytes in 223 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 04:23:37.516665 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c231b2aa-067a-4dda-b41d-294d7a2d291d None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:37.532886 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 454/897] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:23:37 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 25 04:23:37.548452 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:37.622636 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c231b2aa-067a-4dda-b41d-294d7a2d291d None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:37.657044 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f20c7f5-bc50-4994-9c44-f22479d387ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:37.657697 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f20c7f5-bc50-4994-9c44-f22479d387ed None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:37.657978 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f20c7f5-bc50-4994-9c44-f22479d387ed None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:37.658306 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f20c7f5-bc50-4994-9c44-f22479d387ed None admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:37.659000 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f20c7f5-bc50-4994-9c44-f22479d387ed None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:37.659311 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f20c7f5-bc50-4994-9c44-f22479d387ed None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:37.660758 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f20c7f5-bc50-4994-9c44-f22479d387ed None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:37.673770 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 444/898] 10.222.0.25 () {56 vars in 1148 bytes} [Sun Sep 25 04:23:37 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:37.679626 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0ac9426-5d6d-4ed1-b424-b200d5b4c94b None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:37.689643 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f20c7f5-bc50-4994-9c44-f22479d387ed None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:37.771851 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:37.778028 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:37.792230 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f20c7f5-bc50-4994-9c44-f22479d387ed None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:37.795527 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0ac9426-5d6d-4ed1-b424-b200d5b4c94b None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:37.830143 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ee65716-d442-401e-af4c-05ec4838ac10 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:37.830753 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ee65716-d442-401e-af4c-05ec4838ac10 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:37.831030 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ee65716-d442-401e-af4c-05ec4838ac10 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:37.831301 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ee65716-d442-401e-af4c-05ec4838ac10 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:37.832017 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ee65716-d442-401e-af4c-05ec4838ac10 None admin] RBAC: Adding query filter params (role.id=062e3575fa1d40c880fc87ceca94db4a, scope.project.id=056ca1b1a9974a7581550de0d9217823, user.id=050b3aab44bb4b17aa4a1ba7cccdb231) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:37.832323 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ee65716-d442-401e-af4c-05ec4838ac10 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:37.833530 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-558e0c01-30bc-4074-a8f3-aa1a173348bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:37.833777 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ee65716-d442-401e-af4c-05ec4838ac10 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:37.834143 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-558e0c01-30bc-4074-a8f3-aa1a173348bf None admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:37.834427 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-558e0c01-30bc-4074-a8f3-aa1a173348bf None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:37.834701 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-558e0c01-30bc-4074-a8f3-aa1a173348bf None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:37.835629 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-558e0c01-30bc-4074-a8f3-aa1a173348bf None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:37.837700 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-558e0c01-30bc-4074-a8f3-aa1a173348bf None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:37.839412 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 455/899] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Sep 25 04:23:37 2022] GET /identity/v3/role_assignments?user.id=050b3aab44bb4b17aa4a1ba7cccdb231&scope.project.id=056ca1b1a9974a7581550de0d9217823&role.id=062e3575fa1d40c880fc87ceca94db4a => generated 259 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:37.840474 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ee65716-d442-401e-af4c-05ec4838ac10 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:37.864969 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 445/900] 10.222.0.25 () {60 vars in 1141 bytes} [Sun Sep 25 04:23:37 2022] POST /identity/v3/services => generated 241 bytes in 176 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:23:37.979612 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ee65716-d442-401e-af4c-05ec4838ac10 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:38.031000 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fac4ec2b-3e67-4c35-a4c1-9e36bf801ea8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:38.032216 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fac4ec2b-3e67-4c35-a4c1-9e36bf801ea8 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:38.032675 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fac4ec2b-3e67-4c35-a4c1-9e36bf801ea8 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:38.033087 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fac4ec2b-3e67-4c35-a4c1-9e36bf801ea8 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:38.033843 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fac4ec2b-3e67-4c35-a4c1-9e36bf801ea8 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:38.035394 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fac4ec2b-3e67-4c35-a4c1-9e36bf801ea8 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:38.045508 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-fac4ec2b-3e67-4c35-a4c1-9e36bf801ea8 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 25 04:23:38.045508 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:38.045508 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 25 04:23:38.045508 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 25 04:23:38.045508 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 25 04:23:38.045508 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 25 04:23:38.045508 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 25 04:23:38.045508 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 04:23:38.045508 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 25 04:23:38.045508 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:23:38.045508 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 25 04:23:38.045508 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:23:38.045508 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:38.045508 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:38.045508 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:38.045508 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:38.045508 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:38.047381 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:38.047381 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:38.047381 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:38.047381 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:38.047381 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:38.047381 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:38.047381 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 25 04:23:38.047381 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 25 04:23:38.047381 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 25 04:23:38.047381 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 25 04:23:38.047381 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:38.047381 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:38.047381 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:38.047381 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:38.047381 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:38.047381 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:38.048424 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:38.048424 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:38.048424 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:38.048424 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:38.048424 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:38.048424 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:38.048424 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:38.048424 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:38.048424 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:38.048424 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:38.048424 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 25 04:23:38.048424 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 04:23:38.048424 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 25 04:23:38.048424 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:23:38.054900 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 456/901] 10.222.0.25 () {56 vars in 1114 bytes} [Sun Sep 25 04:23:37 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:38.057830 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-558e0c01-30bc-4074-a8f3-aa1a173348bf None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:38.170334 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-558e0c01-30bc-4074-a8f3-aa1a173348bf None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:38.207643 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ee4181c-1581-4046-a121-79034cc81603 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:38.208258 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ee4181c-1581-4046-a121-79034cc81603 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:38.208539 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ee4181c-1581-4046-a121-79034cc81603 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:38.208841 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ee4181c-1581-4046-a121-79034cc81603 None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:38.209469 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ee4181c-1581-4046-a121-79034cc81603 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:38.209773 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ee4181c-1581-4046-a121-79034cc81603 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:38.211151 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ee4181c-1581-4046-a121-79034cc81603 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:38.216336 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 446/902] 10.222.0.25 () {56 vars in 1114 bytes} [Sun Sep 25 04:23:38 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:38.222628 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fac4ec2b-3e67-4c35-a4c1-9e36bf801ea8 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:38.232680 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:38.333231 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fac4ec2b-3e67-4c35-a4c1-9e36bf801ea8 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:38.370163 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e692a34-0c8b-4274-a760-9aee255ee0b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:38.371139 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e692a34-0c8b-4274-a760-9aee255ee0b6 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:38.371616 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e692a34-0c8b-4274-a760-9aee255ee0b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:38.372091 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e692a34-0c8b-4274-a760-9aee255ee0b6 None admin] PATH_INFO: `/v3/services` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:38.372954 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e692a34-0c8b-4274-a760-9aee255ee0b6 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:38.373472 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e692a34-0c8b-4274-a760-9aee255ee0b6 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:38.375471 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e692a34-0c8b-4274-a760-9aee255ee0b6 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:38.384736 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 457/903] 10.222.0.25 () {56 vars in 1114 bytes} [Sun Sep 25 04:23:38 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:38.387362 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ee4181c-1581-4046-a121-79034cc81603 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:38.404539 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:38.505551 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ee4181c-1581-4046-a121-79034cc81603 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:38.542769 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-260400b4-6190-46e9-aa41-60ec06c9fa30 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:38.543630 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-260400b4-6190-46e9-aa41-60ec06c9fa30 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:38.544094 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-260400b4-6190-46e9-aa41-60ec06c9fa30 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:38.544518 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-260400b4-6190-46e9-aa41-60ec06c9fa30 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:38.545396 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-260400b4-6190-46e9-aa41-60ec06c9fa30 None admin] RBAC: Adding query filter params (interface=public, service_id=f3b47a4606a74acb8a173e7ff2299efc, region_id=RegionOne) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:38.546419 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-260400b4-6190-46e9-aa41-60ec06c9fa30 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:38.548113 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-260400b4-6190-46e9-aa41-60ec06c9fa30 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:38.554132 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 447/904] 10.222.0.25 () {56 vars in 1251 bytes} [Sun Sep 25 04:23:38 2022] GET /identity/v3/endpoints?service_id=f3b47a4606a74acb8a173e7ff2299efc&interface=public®ion_id=RegionOne => generated 194 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:38.564701 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e692a34-0c8b-4274-a760-9aee255ee0b6 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:38.634096 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:38.666643 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e692a34-0c8b-4274-a760-9aee255ee0b6 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:38.702267 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50cd62d3-6ee8-4299-aae5-88dc9d9921d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:38.702882 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50cd62d3-6ee8-4299-aae5-88dc9d9921d8 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:38.703161 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50cd62d3-6ee8-4299-aae5-88dc9d9921d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:38.703434 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50cd62d3-6ee8-4299-aae5-88dc9d9921d8 None admin] PATH_INFO: `/v3/services` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:38.704037 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50cd62d3-6ee8-4299-aae5-88dc9d9921d8 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:38.705425 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50cd62d3-6ee8-4299-aae5-88dc9d9921d8 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:38.716785 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 458/905] 10.222.0.25 () {56 vars in 1087 bytes} [Sun Sep 25 04:23:38 2022] GET /identity/v3/services => generated 1497 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:23:38.875115 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d4a218-2f93-4717-9f71-a68ee47b14ba None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:38.875675 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d4a218-2f93-4717-9f71-a68ee47b14ba None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:38.876100 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d4a218-2f93-4717-9f71-a68ee47b14ba None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:38.877901 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 448/906] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:23:38 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:23:38.883590 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7451f1fb-0215-488d-8fcf-8884c35379fe None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:38.883882 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7451f1fb-0215-488d-8fcf-8884c35379fe None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:38.884208 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7451f1fb-0215-488d-8fcf-8884c35379fe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:38.903983 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-7451f1fb-0215-488d-8fcf-8884c35379fe None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:23:38.905722 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-7451f1fb-0215-488d-8fcf-8884c35379fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:23:38.980718 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 459/907] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:23:38 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 04:23:39.250683 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9d4a218-2f93-4717-9f71-a68ee47b14ba None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:39.333314 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:39.359607 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9d4a218-2f93-4717-9f71-a68ee47b14ba None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:39.394885 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d6800aa-d5b8-49e9-8162-b167be32c426 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:39.396322 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d6800aa-d5b8-49e9-8162-b167be32c426 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:39.396830 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d6800aa-d5b8-49e9-8162-b167be32c426 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:39.397329 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d6800aa-d5b8-49e9-8162-b167be32c426 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:39.398105 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d6800aa-d5b8-49e9-8162-b167be32c426 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:39.399678 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d6800aa-d5b8-49e9-8162-b167be32c426 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:39.410085 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-0d6800aa-d5b8-49e9-8162-b167be32c426 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 25 04:23:39.410085 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:39.410085 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 25 04:23:39.410085 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 25 04:23:39.410085 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 25 04:23:39.410085 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 25 04:23:39.410085 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 25 04:23:39.410085 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 04:23:39.410085 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 25 04:23:39.410085 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:39.410085 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 25 04:23:39.410085 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:39.410085 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:39.410085 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:39.410085 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:39.410085 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:39.411074 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:39.411074 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:39.411074 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:39.411074 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:39.411074 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:39.411074 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:39.411074 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:39.411074 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 25 04:23:39.411074 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 25 04:23:39.411074 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 25 04:23:39.411074 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 25 04:23:39.411074 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:39.411074 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:39.411074 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:39.411074 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:39.411074 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:39.411074 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:39.412032 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:39.412032 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:39.412032 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:39.412032 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:39.412032 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:39.412032 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:39.412032 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:39.412032 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:39.412032 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:39.412032 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:39.412032 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 25 04:23:39.412032 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 04:23:39.412032 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 25 04:23:39.412032 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:39.414914 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 449/908] 10.222.0.25 () {56 vars in 1132 bytes} [Sun Sep 25 04:23:39 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:39.422776 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7451f1fb-0215-488d-8fcf-8884c35379fe None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:39.489961 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:39.529368 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7451f1fb-0215-488d-8fcf-8884c35379fe None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:39.565389 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c897b58-1df2-48be-9dc0-d9314d4af72f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:39.566349 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c897b58-1df2-48be-9dc0-d9314d4af72f None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:39.566829 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c897b58-1df2-48be-9dc0-d9314d4af72f None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:39.567300 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c897b58-1df2-48be-9dc0-d9314d4af72f None admin] PATH_INFO: `/v3/services` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:39.568156 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c897b58-1df2-48be-9dc0-d9314d4af72f None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:39.568681 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c897b58-1df2-48be-9dc0-d9314d4af72f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:39.570307 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c897b58-1df2-48be-9dc0-d9314d4af72f None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:39.576138 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 460/909] 10.222.0.25 () {56 vars in 1126 bytes} [Sun Sep 25 04:23:39 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:39.588048 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d6800aa-d5b8-49e9-8162-b167be32c426 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:39.701327 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d6800aa-d5b8-49e9-8162-b167be32c426 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:39.737254 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d03394f2-58ab-44b5-aae5-dc79ef917ad6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:39.737870 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d03394f2-58ab-44b5-aae5-dc79ef917ad6 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:39.738155 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d03394f2-58ab-44b5-aae5-dc79ef917ad6 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:39.738429 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d03394f2-58ab-44b5-aae5-dc79ef917ad6 None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:39.739060 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d03394f2-58ab-44b5-aae5-dc79ef917ad6 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:39.739429 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d03394f2-58ab-44b5-aae5-dc79ef917ad6 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:39.740886 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d03394f2-58ab-44b5-aae5-dc79ef917ad6 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:39.746436 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 450/910] 10.222.0.25 () {56 vars in 1126 bytes} [Sun Sep 25 04:23:39 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:39.752256 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c897b58-1df2-48be-9dc0-d9314d4af72f None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:39.854302 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c897b58-1df2-48be-9dc0-d9314d4af72f None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:39.882117 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:39.890876 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d9e15b6-d1c4-4f9e-b88d-baa664bd8450 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:39.891504 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d9e15b6-d1c4-4f9e-b88d-baa664bd8450 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:39.891787 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d9e15b6-d1c4-4f9e-b88d-baa664bd8450 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:39.892064 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d9e15b6-d1c4-4f9e-b88d-baa664bd8450 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:39.892822 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d9e15b6-d1c4-4f9e-b88d-baa664bd8450 None admin] RBAC: Adding query filter params (interface=public, service_id=8f5fdaed542e409da3457181822466a7, region_id=RegionOne) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:39.893451 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d9e15b6-d1c4-4f9e-b88d-baa664bd8450 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:39.894931 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d9e15b6-d1c4-4f9e-b88d-baa664bd8450 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:39.901412 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 461/911] 10.222.0.25 () {56 vars in 1251 bytes} [Sun Sep 25 04:23:39 2022] GET /identity/v3/endpoints?service_id=8f5fdaed542e409da3457181822466a7&interface=public®ion_id=RegionOne => generated 194 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:39.907524 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d03394f2-58ab-44b5-aae5-dc79ef917ad6 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:39.911968 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:40.020043 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d03394f2-58ab-44b5-aae5-dc79ef917ad6 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:40.020705 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc5f841d-de38-4260-8f0c-09e49e4a0c30 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:40.021159 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc5f841d-de38-4260-8f0c-09e49e4a0c30 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:40.021240 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc5f841d-de38-4260-8f0c-09e49e4a0c30 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:40.022799 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 462/912] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:23:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:23:40.027909 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6714474d-472a-489e-9d81-ca0c5b70ebda None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:40.028186 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6714474d-472a-489e-9d81-ca0c5b70ebda None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:40.028454 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6714474d-472a-489e-9d81-ca0c5b70ebda None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:40.050619 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-6714474d-472a-489e-9d81-ca0c5b70ebda None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:23:40.051849 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-6714474d-472a-489e-9d81-ca0c5b70ebda None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:23:40.056385 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d24cacaf-1f2c-47ab-91c2-6e789dfe15bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:40.057029 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d24cacaf-1f2c-47ab-91c2-6e789dfe15bd None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:40.057312 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d24cacaf-1f2c-47ab-91c2-6e789dfe15bd None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:40.057590 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d24cacaf-1f2c-47ab-91c2-6e789dfe15bd None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:40.058195 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d24cacaf-1f2c-47ab-91c2-6e789dfe15bd None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:40.059561 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d24cacaf-1f2c-47ab-91c2-6e789dfe15bd None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:40.066860 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 451/913] 10.222.0.25 () {56 vars in 1087 bytes} [Sun Sep 25 04:23:39 2022] GET /identity/v3/services => generated 1497 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:23:40.130313 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 463/914] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:23:40 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 25 04:23:40.332828 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bebe0447-0632-4c05-b7c5-c759e209fbf3 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:40.333132 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bebe0447-0632-4c05-b7c5-c759e209fbf3 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:40.333566 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bebe0447-0632-4c05-b7c5-c759e209fbf3 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:40.335223 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 452/915] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:23:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:23:40.340709 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01963f67-308c-4ed4-bd7b-55ebf27d063c None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:40.341096 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01963f67-308c-4ed4-bd7b-55ebf27d063c None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:40.341445 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01963f67-308c-4ed4-bd7b-55ebf27d063c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:40.360722 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-01963f67-308c-4ed4-bd7b-55ebf27d063c None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:23:40.361922 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-01963f67-308c-4ed4-bd7b-55ebf27d063c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:23:40.398804 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bebe0447-0632-4c05-b7c5-c759e209fbf3 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:40.436746 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 464/916] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:23:40 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 04:23:40.493912 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:40.537948 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bebe0447-0632-4c05-b7c5-c759e209fbf3 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:40.574324 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-add7a43b-4a5d-48df-a5ac-87ba0a421f66 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:40.575373 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add7a43b-4a5d-48df-a5ac-87ba0a421f66 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:40.576113 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add7a43b-4a5d-48df-a5ac-87ba0a421f66 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:40.576605 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add7a43b-4a5d-48df-a5ac-87ba0a421f66 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:40.588764 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-add7a43b-4a5d-48df-a5ac-87ba0a421f66 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 04:23:40.594173 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-add7a43b-4a5d-48df-a5ac-87ba0a421f66 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 25 04:23:40.594173 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:40.594173 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:40.594173 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:40.594173 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:40.594173 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:40.594173 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:40.594173 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:40.594173 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:40.594173 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:40.594173 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:40.594173 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:40.594173 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 04:23:40.594173 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 04:23:40.594173 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 04:23:40.594173 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 04:23:40.595136 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 04:23:40.595136 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 04:23:40.595136 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 04:23:40.595136 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 04:23:40.595136 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:40.595136 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:40.595136 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:40.595136 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:40.595136 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:40.595136 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:40.595136 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:40.595136 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:40.595136 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:40.595136 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:40.595136 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:40.595136 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:40.596002 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:40.596002 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:40.596002 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:40.596002 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:40.596002 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 04:23:40.596002 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 04:23:40.596002 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 04:23:40.596002 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 04:23:40.596002 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 04:23:40.596002 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 04:23:40.596002 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 25 04:23:40.596002 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:40.598696 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 453/917] 10.222.0.25 () {56 vars in 1120 bytes} [Sun Sep 25 04:23:40 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:40.606925 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01963f67-308c-4ed4-bd7b-55ebf27d063c None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:40.681135 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:40.711468 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01963f67-308c-4ed4-bd7b-55ebf27d063c None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:40.732836 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-add7a43b-4a5d-48df-a5ac-87ba0a421f66 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:40.751308 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9d88698-bae7-45f2-ba43-d4edf7ea7cb5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:40.752284 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9d88698-bae7-45f2-ba43-d4edf7ea7cb5 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:40.752842 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9d88698-bae7-45f2-ba43-d4edf7ea7cb5 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:40.753596 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9d88698-bae7-45f2-ba43-d4edf7ea7cb5 None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:40.754957 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9d88698-bae7-45f2-ba43-d4edf7ea7cb5 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:40.755468 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9d88698-bae7-45f2-ba43-d4edf7ea7cb5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:40.757628 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9d88698-bae7-45f2-ba43-d4edf7ea7cb5 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:40.768391 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 465/918] 10.222.0.25 () {56 vars in 1115 bytes} [Sun Sep 25 04:23:40 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:40.773511 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9d88698-bae7-45f2-ba43-d4edf7ea7cb5 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:40.848831 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-add7a43b-4a5d-48df-a5ac-87ba0a421f66 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:40.881093 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:40.890754 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cbc8bf7-1f93-46b9-b387-51dc1313ca77 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:40.891349 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cbc8bf7-1f93-46b9-b387-51dc1313ca77 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:40.891636 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cbc8bf7-1f93-46b9-b387-51dc1313ca77 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:40.891911 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cbc8bf7-1f93-46b9-b387-51dc1313ca77 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:40.892498 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cbc8bf7-1f93-46b9-b387-51dc1313ca77 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:40.893932 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cbc8bf7-1f93-46b9-b387-51dc1313ca77 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:40.903699 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-2cbc8bf7-1f93-46b9-b387-51dc1313ca77 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 25 04:23:40.903699 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:40.903699 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 25 04:23:40.903699 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 25 04:23:40.903699 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 25 04:23:40.903699 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 25 04:23:40.903699 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 25 04:23:40.903699 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 04:23:40.903699 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 25 04:23:40.903699 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:40.903699 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 25 04:23:40.903699 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:40.903699 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:40.903699 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:40.903699 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:40.903699 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:40.903699 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:40.904727 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:40.904727 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:40.904727 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:40.904727 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:40.904727 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:40.904727 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:40.904727 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 25 04:23:40.904727 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 25 04:23:40.904727 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 25 04:23:40.904727 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 25 04:23:40.904727 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:40.904727 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:40.904727 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:40.904727 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:40.904727 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:40.904727 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:40.906206 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:40.906206 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:40.906206 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:40.906206 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:40.906206 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:40.906206 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:40.906206 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:40.906206 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:40.906206 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:40.906206 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:40.906206 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 25 04:23:40.906206 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 04:23:40.906206 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 25 04:23:40.906206 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:40.908449 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9d88698-bae7-45f2-ba43-d4edf7ea7cb5 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:40.908976 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 454/919] 10.222.0.25 () {56 vars in 1126 bytes} [Sun Sep 25 04:23:40 2022] GET /identity/v3/services/object-store => generated 93 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:40.912224 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cbc8bf7-1f93-46b9-b387-51dc1313ca77 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:40.927180 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:40.951620 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0e5b69b-c8ab-4dc0-b34b-92c35bdd1e9c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:40.953775 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0e5b69b-c8ab-4dc0-b34b-92c35bdd1e9c None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:40.954240 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0e5b69b-c8ab-4dc0-b34b-92c35bdd1e9c None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:40.954659 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0e5b69b-c8ab-4dc0-b34b-92c35bdd1e9c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:40.968374 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0e5b69b-c8ab-4dc0-b34b-92c35bdd1e9c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:40.970399 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0e5b69b-c8ab-4dc0-b34b-92c35bdd1e9c None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:40.988076 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-f0e5b69b-c8ab-4dc0-b34b-92c35bdd1e9c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:23:40.988076 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:40.988076 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:40.988076 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:40.988076 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:40.988076 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:40.988076 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:40.988076 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:40.988076 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:40.988076 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:40.988076 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:40.988076 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:40.988076 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 04:23:40.988076 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 04:23:40.988076 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 04:23:40.988076 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 04:23:40.989478 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:40.989478 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:40.989478 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:40.989478 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:40.989478 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:40.989478 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:40.989478 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:40.989478 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:40.989478 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:40.989478 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:40.989478 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:40.989478 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:40.989478 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:40.989478 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:40.989478 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:40.989478 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:40.989478 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 04:23:40.990593 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 04:23:40.990593 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:23:40.990593 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:23:40.990593 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 466/920] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:23:40 2022] GET /identity/v3/domains/Default => generated 87 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 04:23:40.990878 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31476f0a-cab6-4ae3-b6d5-e7f8b3a8a54b None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:40.991223 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31476f0a-cab6-4ae3-b6d5-e7f8b3a8a54b None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:40.991329 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31476f0a-cab6-4ae3-b6d5-e7f8b3a8a54b None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:40.996345 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 467/921] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:23:40 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:23:40.997225 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31476f0a-cab6-4ae3-b6d5-e7f8b3a8a54b None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:41.033699 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cbc8bf7-1f93-46b9-b387-51dc1313ca77 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:41.072356 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b47bcfac-c66a-45c3-b70c-3a1d7b653fba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:41.072991 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47bcfac-c66a-45c3-b70c-3a1d7b653fba None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:41.073311 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47bcfac-c66a-45c3-b70c-3a1d7b653fba None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:41.073533 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47bcfac-c66a-45c3-b70c-3a1d7b653fba None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:41.074148 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b47bcfac-c66a-45c3-b70c-3a1d7b653fba None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:41.074447 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b47bcfac-c66a-45c3-b70c-3a1d7b653fba None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:41.076157 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b47bcfac-c66a-45c3-b70c-3a1d7b653fba None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:41.081534 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 455/922] 10.222.0.25 () {56 vars in 1122 bytes} [Sun Sep 25 04:23:40 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:41.082523 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3d80c96-aa2d-4181-ab79-31c8ee199f60 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:41.082811 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3d80c96-aa2d-4181-ab79-31c8ee199f60 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:41.083079 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3d80c96-aa2d-4181-ab79-31c8ee199f60 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:41.102265 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-d3d80c96-aa2d-4181-ab79-31c8ee199f60 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:23:41.103465 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-d3d80c96-aa2d-4181-ab79-31c8ee199f60 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:23:41.110183 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31476f0a-cab6-4ae3-b6d5-e7f8b3a8a54b None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:41.145564 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be160d90-fcbb-46a5-97f4-2a07b3175ada None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:41.146176 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be160d90-fcbb-46a5-97f4-2a07b3175ada None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:41.146473 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be160d90-fcbb-46a5-97f4-2a07b3175ada None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:41.146729 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be160d90-fcbb-46a5-97f4-2a07b3175ada None admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:41.147363 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be160d90-fcbb-46a5-97f4-2a07b3175ada None admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:41.147674 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be160d90-fcbb-46a5-97f4-2a07b3175ada None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:41.149076 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be160d90-fcbb-46a5-97f4-2a07b3175ada None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:41.156317 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:41.161086 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 468/923] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Sep 25 04:23:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:41.161897 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be160d90-fcbb-46a5-97f4-2a07b3175ada None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:41.180090 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 456/924] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:23:41 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 04:23:41.181444 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3d80c96-aa2d-4181-ab79-31c8ee199f60 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:41.192313 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:41.274894 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be160d90-fcbb-46a5-97f4-2a07b3175ada None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:41.310958 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3d80c96-aa2d-4181-ab79-31c8ee199f60 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:41.315459 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e28b31e8-71f4-40c0-ad6d-a71aeaef7977 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:41.316332 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e28b31e8-71f4-40c0-ad6d-a71aeaef7977 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:41.316823 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e28b31e8-71f4-40c0-ad6d-a71aeaef7977 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:41.317271 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e28b31e8-71f4-40c0-ad6d-a71aeaef7977 None admin] PATH_INFO: `/v3/services` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:41.318097 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e28b31e8-71f4-40c0-ad6d-a71aeaef7977 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:41.318712 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e28b31e8-71f4-40c0-ad6d-a71aeaef7977 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:41.320087 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e28b31e8-71f4-40c0-ad6d-a71aeaef7977 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:41.328691 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 469/925] 10.222.0.25 () {56 vars in 1122 bytes} [Sun Sep 25 04:23:41 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:41.332759 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e28b31e8-71f4-40c0-ad6d-a71aeaef7977 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:41.354520 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-658f56a0-7049-4a59-80ee-8af3dacee2a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:41.355612 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-658f56a0-7049-4a59-80ee-8af3dacee2a4 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:41.356538 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-658f56a0-7049-4a59-80ee-8af3dacee2a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:41.357070 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-658f56a0-7049-4a59-80ee-8af3dacee2a4 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:41.367797 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-658f56a0-7049-4a59-80ee-8af3dacee2a4 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:41.370822 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-658f56a0-7049-4a59-80ee-8af3dacee2a4 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:41.387559 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-658f56a0-7049-4a59-80ee-8af3dacee2a4 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 25 04:23:41.387559 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:41.387559 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:41.387559 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:41.387559 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:41.387559 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:41.387559 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:41.387559 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:41.387559 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:41.387559 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:41.387559 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:41.387559 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:41.387559 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 25 04:23:41.387559 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 25 04:23:41.387559 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 25 04:23:41.387559 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 25 04:23:41.388603 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:41.388603 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:41.388603 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 25 04:23:41.388603 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 04:23:41.388603 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 25 04:23:41.388603 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 04:23:41.388603 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:41.388603 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:41.388603 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:41.388603 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:41.388603 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:41.388603 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:41.388603 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:41.388603 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:41.388603 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:41.388603 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:41.388603 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:41.388603 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:41.390375 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:41.390375 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:41.390375 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 25 04:23:41.390375 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 25 04:23:41.390375 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 25 04:23:41.390375 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 25 04:23:41.390375 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 25 04:23:41.390375 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 25 04:23:41.390375 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 25 04:23:41.390375 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:41.393387 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 457/926] 10.222.0.25 () {56 vars in 1117 bytes} [Sun Sep 25 04:23:41 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:41.401682 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-658f56a0-7049-4a59-80ee-8af3dacee2a4 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:41.437869 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:41.449415 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e28b31e8-71f4-40c0-ad6d-a71aeaef7977 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:41.459863 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:41.495300 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cb25b46-a76f-4c79-8963-98f3bd17ed52 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:41.495906 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb25b46-a76f-4c79-8963-98f3bd17ed52 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:41.496184 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb25b46-a76f-4c79-8963-98f3bd17ed52 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:41.496450 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb25b46-a76f-4c79-8963-98f3bd17ed52 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:41.497173 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cb25b46-a76f-4c79-8963-98f3bd17ed52 None admin] RBAC: Adding query filter params (interface=public, service_id=fae12ee1be974fce89b644eeb639d0c4, region_id=RegionOne) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:41.497480 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cb25b46-a76f-4c79-8963-98f3bd17ed52 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:41.499373 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cb25b46-a76f-4c79-8963-98f3bd17ed52 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:41.511585 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 470/927] 10.222.0.25 () {56 vars in 1251 bytes} [Sun Sep 25 04:23:41 2022] GET /identity/v3/endpoints?service_id=fae12ee1be974fce89b644eeb639d0c4&interface=public®ion_id=RegionOne => generated 194 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:41.511585 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cb25b46-a76f-4c79-8963-98f3bd17ed52 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:41.532174 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-658f56a0-7049-4a59-80ee-8af3dacee2a4 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:41.567079 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59fbbb8d-ba98-4d78-a2e5-51ccdbcc37f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:41.567799 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59fbbb8d-ba98-4d78-a2e5-51ccdbcc37f8 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:41.568076 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59fbbb8d-ba98-4d78-a2e5-51ccdbcc37f8 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:41.568455 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59fbbb8d-ba98-4d78-a2e5-51ccdbcc37f8 None admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:41.569205 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59fbbb8d-ba98-4d78-a2e5-51ccdbcc37f8 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:41.569508 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59fbbb8d-ba98-4d78-a2e5-51ccdbcc37f8 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:41.571019 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59fbbb8d-ba98-4d78-a2e5-51ccdbcc37f8 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:41.594629 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 458/928] 10.222.0.25 () {56 vars in 1149 bytes} [Sun Sep 25 04:23:41 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:41.595497 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59fbbb8d-ba98-4d78-a2e5-51ccdbcc37f8 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:41.610976 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cb25b46-a76f-4c79-8963-98f3bd17ed52 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:41.645931 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55b08331-c886-4a71-919e-bb8fe92cff12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:41.646567 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55b08331-c886-4a71-919e-bb8fe92cff12 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:41.646845 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55b08331-c886-4a71-919e-bb8fe92cff12 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:41.647116 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55b08331-c886-4a71-919e-bb8fe92cff12 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:41.647691 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55b08331-c886-4a71-919e-bb8fe92cff12 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:41.649095 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55b08331-c886-4a71-919e-bb8fe92cff12 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:41.658583 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-55b08331-c886-4a71-919e-bb8fe92cff12 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 25 04:23:41.658583 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:41.658583 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 25 04:23:41.658583 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 25 04:23:41.658583 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 25 04:23:41.658583 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 25 04:23:41.658583 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 25 04:23:41.658583 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 04:23:41.658583 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 25 04:23:41.658583 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:23:41.658583 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 25 04:23:41.658583 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:23:41.658583 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:41.658583 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:41.658583 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:41.658583 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:41.658583 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:41.659589 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:41.659589 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:41.659589 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:41.659589 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:41.659589 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:41.659589 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:41.659589 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 25 04:23:41.659589 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 25 04:23:41.659589 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 25 04:23:41.659589 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 25 04:23:41.659589 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:41.659589 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:41.659589 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:41.659589 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:41.659589 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:41.659589 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:41.660457 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:41.660457 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:41.660457 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:41.660457 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:41.660457 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:41.660457 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:41.660457 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:41.660457 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:41.660457 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:41.660457 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:41.660457 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 25 04:23:41.660457 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 04:23:41.660457 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 25 04:23:41.660457 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:23:41.660457 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 471/929] 10.222.0.25 () {56 vars in 1114 bytes} [Sun Sep 25 04:23:41 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:41.661301 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55b08331-c886-4a71-919e-bb8fe92cff12 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:41.687724 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:41.701555 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59fbbb8d-ba98-4d78-a2e5-51ccdbcc37f8 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:41.704068 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:41.737847 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ef70395-beec-4514-9864-d26e2cc3d0ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:41.738456 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ef70395-beec-4514-9864-d26e2cc3d0ff None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:41.738747 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ef70395-beec-4514-9864-d26e2cc3d0ff None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:41.739034 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ef70395-beec-4514-9864-d26e2cc3d0ff None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:41.739598 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ef70395-beec-4514-9864-d26e2cc3d0ff None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:41.740990 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ef70395-beec-4514-9864-d26e2cc3d0ff None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:41.746794 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 459/930] 10.222.0.25 () {56 vars in 1087 bytes} [Sun Sep 25 04:23:41 2022] GET /identity/v3/services => generated 1497 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:23:41.746794 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ef70395-beec-4514-9864-d26e2cc3d0ff None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:41.785852 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55b08331-c886-4a71-919e-bb8fe92cff12 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:41.826607 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6b8c627-e097-46eb-968f-054fc11f378c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:41.827799 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6b8c627-e097-46eb-968f-054fc11f378c None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:41.828078 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6b8c627-e097-46eb-968f-054fc11f378c None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:41.828341 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6b8c627-e097-46eb-968f-054fc11f378c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:41.841530 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6b8c627-e097-46eb-968f-054fc11f378c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:41.843229 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6b8c627-e097-46eb-968f-054fc11f378c None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:41.860613 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-d6b8c627-e097-46eb-968f-054fc11f378c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:23:41.860613 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:41.860613 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:41.860613 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:41.860613 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:41.860613 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:41.860613 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:41.860613 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:41.860613 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:41.860613 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:41.860613 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:41.860613 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:41.860613 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 04:23:41.860613 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 04:23:41.860613 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 04:23:41.860613 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 04:23:41.861681 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:41.861681 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:41.861681 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:41.861681 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:41.861681 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:41.861681 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:41.861681 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:41.861681 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:41.861681 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:41.861681 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:41.861681 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:41.861681 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:41.861681 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:41.861681 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:41.861681 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:41.861681 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:41.861681 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 04:23:41.862602 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 04:23:41.862602 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:23:41.862602 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:23:41.866797 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 472/931] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:23:41 2022] GET /identity/v3/domains/Default => generated 87 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:41.868118 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6b8c627-e097-46eb-968f-054fc11f378c None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:41.900261 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ef70395-beec-4514-9864-d26e2cc3d0ff None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:41.942742 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afd0af54-69cb-4143-acf8-f1dd42deb1a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:41.943460 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd0af54-69cb-4143-acf8-f1dd42deb1a8 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:41.943831 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd0af54-69cb-4143-acf8-f1dd42deb1a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:41.944297 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd0af54-69cb-4143-acf8-f1dd42deb1a8 None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:41.945012 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afd0af54-69cb-4143-acf8-f1dd42deb1a8 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:41.945615 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afd0af54-69cb-4143-acf8-f1dd42deb1a8 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:41.947855 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afd0af54-69cb-4143-acf8-f1dd42deb1a8 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:41.955720 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 460/932] 10.222.0.25 () {56 vars in 1114 bytes} [Sun Sep 25 04:23:41 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:41.958103 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:41.962028 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afd0af54-69cb-4143-acf8-f1dd42deb1a8 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:42.004156 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:42.006047 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6b8c627-e097-46eb-968f-054fc11f378c None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:42.042238 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e286f55-4eef-41bc-b2a3-7b243e34ff2f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:42.042804 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e286f55-4eef-41bc-b2a3-7b243e34ff2f None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:42.043035 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e286f55-4eef-41bc-b2a3-7b243e34ff2f None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:42.043300 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e286f55-4eef-41bc-b2a3-7b243e34ff2f None admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:42.043996 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e286f55-4eef-41bc-b2a3-7b243e34ff2f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:42.044294 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e286f55-4eef-41bc-b2a3-7b243e34ff2f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:42.046560 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e286f55-4eef-41bc-b2a3-7b243e34ff2f None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:42.059576 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 473/933] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Sep 25 04:23:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:23:42.065062 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e286f55-4eef-41bc-b2a3-7b243e34ff2f None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:42.101831 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afd0af54-69cb-4143-acf8-f1dd42deb1a8 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:42.139768 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b5ce70c-028a-4e62-b8cc-eaa6c10cfeff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:42.140466 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b5ce70c-028a-4e62-b8cc-eaa6c10cfeff None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:42.140763 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b5ce70c-028a-4e62-b8cc-eaa6c10cfeff None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:42.141035 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b5ce70c-028a-4e62-b8cc-eaa6c10cfeff None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:42.141659 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b5ce70c-028a-4e62-b8cc-eaa6c10cfeff None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:42.142017 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b5ce70c-028a-4e62-b8cc-eaa6c10cfeff None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:42.143389 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b5ce70c-028a-4e62-b8cc-eaa6c10cfeff None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:42.153155 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 461/934] 10.222.0.25 () {56 vars in 1114 bytes} [Sun Sep 25 04:23:41 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:42.154689 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b5ce70c-028a-4e62-b8cc-eaa6c10cfeff None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:42.195085 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e286f55-4eef-41bc-b2a3-7b243e34ff2f None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:42.232932 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:42.241688 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76330cd8-1fbe-40e2-b00f-5c241028e4aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:42.242562 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76330cd8-1fbe-40e2-b00f-5c241028e4aa None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:42.243007 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76330cd8-1fbe-40e2-b00f-5c241028e4aa None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:42.243582 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76330cd8-1fbe-40e2-b00f-5c241028e4aa None admin] PATH_INFO: `/v3/projects/service` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:42.253651 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76330cd8-1fbe-40e2-b00f-5c241028e4aa None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:42.255385 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76330cd8-1fbe-40e2-b00f-5c241028e4aa None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:42.270581 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-76330cd8-1fbe-40e2-b00f-5c241028e4aa None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 04:23:42.270581 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:42.270581 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:42.270581 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:42.270581 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:42.270581 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:42.270581 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:42.270581 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:42.270581 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:42.270581 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:42.270581 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:42.270581 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:42.270581 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 25 04:23:42.270581 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 25 04:23:42.270581 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 25 04:23:42.271726 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 25 04:23:42.271726 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:42.271726 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:42.271726 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:42.271726 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:42.271726 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:42.271726 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:42.271726 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:42.271726 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:42.271726 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:42.271726 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:42.271726 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:42.271726 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:42.271726 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:42.271726 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:42.271726 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:42.271726 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:42.272884 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 25 04:23:42.272884 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 25 04:23:42.272884 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 25 04:23:42.272884 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 25 04:23:42.272884 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 25 04:23:42.272884 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 25 04:23:42.272884 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 04:23:42.272884 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:23:42.275347 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 474/935] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Sep 25 04:23:42 2022] GET /identity/v3/projects/service => generated 88 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 04:23:42.281543 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76330cd8-1fbe-40e2-b00f-5c241028e4aa None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:42.284492 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:42.291649 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b5ce70c-028a-4e62-b8cc-eaa6c10cfeff None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:42.330131 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-464f6193-c744-49b8-850f-4a7098ed56ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:42.330662 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-464f6193-c744-49b8-850f-4a7098ed56ba None admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:42.330942 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-464f6193-c744-49b8-850f-4a7098ed56ba None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:42.331213 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-464f6193-c744-49b8-850f-4a7098ed56ba None admin] PATH_INFO: `/v3/endpoints` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:42.332196 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-464f6193-c744-49b8-850f-4a7098ed56ba None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:42.333575 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-464f6193-c744-49b8-850f-4a7098ed56ba None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:42.361519 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 462/936] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:23:42 2022] POST /identity/v3/endpoints => generated 346 bytes in 203 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 04:23:42.361519 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3181027-e555-418e-8335-e20bcafcd4d6 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:42.361519 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3181027-e555-418e-8335-e20bcafcd4d6 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:42.361519 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3181027-e555-418e-8335-e20bcafcd4d6 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:42.361519 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 463/937] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:23:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:23:42.366182 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b24f0fa-0e86-44b4-ba8f-7e768b7ddd42 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:42.366464 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b24f0fa-0e86-44b4-ba8f-7e768b7ddd42 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:42.366733 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b24f0fa-0e86-44b4-ba8f-7e768b7ddd42 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:42.395853 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-1b24f0fa-0e86-44b4-ba8f-7e768b7ddd42 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:23:42.397552 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-1b24f0fa-0e86-44b4-ba8f-7e768b7ddd42 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:23:42.409661 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76330cd8-1fbe-40e2-b00f-5c241028e4aa None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:42.458553 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c5e4ba7-db11-4fe7-909d-73d8eda52fb6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:42.459258 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c5e4ba7-db11-4fe7-909d-73d8eda52fb6 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:42.459580 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c5e4ba7-db11-4fe7-909d-73d8eda52fb6 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:42.459853 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c5e4ba7-db11-4fe7-909d-73d8eda52fb6 None admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:42.460685 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c5e4ba7-db11-4fe7-909d-73d8eda52fb6 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:42.461048 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c5e4ba7-db11-4fe7-909d-73d8eda52fb6 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:42.462549 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c5e4ba7-db11-4fe7-909d-73d8eda52fb6 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:42.489003 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 475/938] 10.222.0.25 () {56 vars in 1148 bytes} [Sun Sep 25 04:23:42 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:42.490944 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c5e4ba7-db11-4fe7-909d-73d8eda52fb6 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:42.504740 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 464/939] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:23:42 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 25 04:23:42.553216 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:42.612241 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c5e4ba7-db11-4fe7-909d-73d8eda52fb6 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:42.647228 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0df7917e-88e6-4c44-aeb6-9e0a0e66195b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:42.647880 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0df7917e-88e6-4c44-aeb6-9e0a0e66195b None admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:42.648160 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0df7917e-88e6-4c44-aeb6-9e0a0e66195b None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:42.648430 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0df7917e-88e6-4c44-aeb6-9e0a0e66195b None admin] PATH_INFO: `/v3/projects/056ca1b1a9974a7581550de0d9217823/users/050b3aab44bb4b17aa4a1ba7cccdb231/roles/062e3575fa1d40c880fc87ceca94db4a` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:42.650649 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0df7917e-88e6-4c44-aeb6-9e0a0e66195b None admin] RBAC: Authorizing `identity:create_grant(project_id=056ca1b1a9974a7581550de0d9217823, user_id=050b3aab44bb4b17aa4a1ba7cccdb231, role_id=062e3575fa1d40c880fc87ceca94db4a)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:42.652074 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0df7917e-88e6-4c44-aeb6-9e0a0e66195b None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:42.669278 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 476/940] 10.222.0.25 () {58 vars in 1439 bytes} [Sun Sep 25 04:23:42 2022] PUT /identity/v3/projects/056ca1b1a9974a7581550de0d9217823/users/050b3aab44bb4b17aa4a1ba7cccdb231/roles/062e3575fa1d40c880fc87ceca94db4a => generated 0 bytes in 179 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 04:23:42.790973 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b24f0fa-0e86-44b4-ba8f-7e768b7ddd42 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:42.881511 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:42.902285 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b24f0fa-0e86-44b4-ba8f-7e768b7ddd42 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:42.940682 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-345f9683-99e3-4dda-87e6-a1aab5d5dad8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:42.940682 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-345f9683-99e3-4dda-87e6-a1aab5d5dad8 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:42.940682 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-345f9683-99e3-4dda-87e6-a1aab5d5dad8 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:42.940682 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-345f9683-99e3-4dda-87e6-a1aab5d5dad8 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:42.941094 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-345f9683-99e3-4dda-87e6-a1aab5d5dad8 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:42.941353 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-345f9683-99e3-4dda-87e6-a1aab5d5dad8 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:42.950217 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-345f9683-99e3-4dda-87e6-a1aab5d5dad8 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 25 04:23:42.950217 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:42.950217 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 25 04:23:42.950217 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 25 04:23:42.950217 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 25 04:23:42.950217 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 25 04:23:42.950217 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 25 04:23:42.950217 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 04:23:42.950217 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 25 04:23:42.950217 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:42.950217 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 25 04:23:42.950217 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:42.950217 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:42.950217 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:42.950217 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:42.950217 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:42.951444 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:42.951444 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:42.951444 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:42.951444 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:42.951444 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:42.951444 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:42.951444 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:42.951444 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 25 04:23:42.951444 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 25 04:23:42.951444 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 25 04:23:42.951444 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 25 04:23:42.951444 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:42.951444 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:42.951444 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:42.951444 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:42.951444 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:42.951444 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:42.953104 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:42.953104 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:42.953104 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:42.953104 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:42.953104 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:42.953104 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:42.953104 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:42.953104 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:42.953104 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:42.953104 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:42.953104 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 25 04:23:42.953104 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 04:23:42.953104 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 25 04:23:42.953104 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:42.953104 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 465/941] 10.222.0.25 () {56 vars in 1132 bytes} [Sun Sep 25 04:23:42 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:42.957777 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0df7917e-88e6-4c44-aeb6-9e0a0e66195b None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:43.059162 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0df7917e-88e6-4c44-aeb6-9e0a0e66195b None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:43.067882 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:43.099836 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34930004-8fcd-41cd-a1e7-66c3704fb42d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:43.100450 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34930004-8fcd-41cd-a1e7-66c3704fb42d None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:43.100752 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34930004-8fcd-41cd-a1e7-66c3704fb42d None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:43.101019 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34930004-8fcd-41cd-a1e7-66c3704fb42d None admin] PATH_INFO: `/v3/services` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:43.101645 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34930004-8fcd-41cd-a1e7-66c3704fb42d None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:43.102028 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34930004-8fcd-41cd-a1e7-66c3704fb42d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:43.103396 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34930004-8fcd-41cd-a1e7-66c3704fb42d None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:43.109095 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 477/942] 10.222.0.25 () {56 vars in 1126 bytes} [Sun Sep 25 04:23:42 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:23:43.114321 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-345f9683-99e3-4dda-87e6-a1aab5d5dad8 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:43.225100 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-345f9683-99e3-4dda-87e6-a1aab5d5dad8 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:43.261098 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3af05b10-9015-4e35-8c9f-9ac2225454d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:43.262057 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3af05b10-9015-4e35-8c9f-9ac2225454d8 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:43.262579 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3af05b10-9015-4e35-8c9f-9ac2225454d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:43.263052 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3af05b10-9015-4e35-8c9f-9ac2225454d8 None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:43.263876 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3af05b10-9015-4e35-8c9f-9ac2225454d8 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:43.264971 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3af05b10-9015-4e35-8c9f-9ac2225454d8 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:43.266540 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3af05b10-9015-4e35-8c9f-9ac2225454d8 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:43.272066 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 466/943] 10.222.0.25 () {56 vars in 1126 bytes} [Sun Sep 25 04:23:43 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:43.277473 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34930004-8fcd-41cd-a1e7-66c3704fb42d None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:43.386479 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34930004-8fcd-41cd-a1e7-66c3704fb42d None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:43.423834 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64dab19f-0104-4173-826c-fad560427a16 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:43.424447 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64dab19f-0104-4173-826c-fad560427a16 None admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:43.424751 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64dab19f-0104-4173-826c-fad560427a16 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:43.425022 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64dab19f-0104-4173-826c-fad560427a16 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:43.425697 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64dab19f-0104-4173-826c-fad560427a16 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:43.427051 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64dab19f-0104-4173-826c-fad560427a16 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:43.449141 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 478/944] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:23:43 2022] POST /identity/v3/endpoints => generated 347 bytes in 172 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:23:44.084224 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f255d58-a4d6-498f-b43a-7e8eddb80472 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:44.084777 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f255d58-a4d6-498f-b43a-7e8eddb80472 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:44.084777 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f255d58-a4d6-498f-b43a-7e8eddb80472 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:44.086355 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 467/945] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:23:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:23:44.092477 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb51c28-04b9-4dac-bc2e-a612c8dae236 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:44.093023 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb51c28-04b9-4dac-bc2e-a612c8dae236 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:44.093525 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb51c28-04b9-4dac-bc2e-a612c8dae236 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:44.113023 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-adb51c28-04b9-4dac-bc2e-a612c8dae236 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:23:44.114207 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-adb51c28-04b9-4dac-bc2e-a612c8dae236 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:23:44.136462 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:44.193877 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 479/946] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:23:44 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 25 04:23:44.481956 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f255d58-a4d6-498f-b43a-7e8eddb80472 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:44.495190 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:44.589993 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f255d58-a4d6-498f-b43a-7e8eddb80472 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:44.627795 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0ec486d-c527-493f-af6d-c06d0236ad81 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:44.628439 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0ec486d-c527-493f-af6d-c06d0236ad81 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:44.628745 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0ec486d-c527-493f-af6d-c06d0236ad81 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:44.629017 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0ec486d-c527-493f-af6d-c06d0236ad81 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:44.629600 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0ec486d-c527-493f-af6d-c06d0236ad81 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:44.630972 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0ec486d-c527-493f-af6d-c06d0236ad81 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:44.640811 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-d0ec486d-c527-493f-af6d-c06d0236ad81 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 25 04:23:44.640811 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:44.640811 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 25 04:23:44.640811 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 25 04:23:44.640811 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 25 04:23:44.640811 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 25 04:23:44.640811 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 25 04:23:44.640811 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 04:23:44.640811 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 25 04:23:44.640811 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:44.640811 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 25 04:23:44.640811 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:44.640811 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:44.640811 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:44.640811 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:44.640811 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:44.640811 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:44.641840 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:44.641840 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:44.641840 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:44.641840 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:44.641840 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:44.641840 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:44.641840 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 25 04:23:44.641840 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 25 04:23:44.641840 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 25 04:23:44.641840 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 25 04:23:44.641840 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:44.641840 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:44.641840 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:44.641840 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:44.641840 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:44.641840 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:44.642691 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:44.642691 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:44.642691 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:44.642691 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:44.642691 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:44.642691 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:44.642691 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:44.642691 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:44.642691 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:44.642691 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:44.642691 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 25 04:23:44.642691 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 04:23:44.642691 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 25 04:23:44.642691 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:44.642691 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 468/947] 10.222.0.25 () {56 vars in 1126 bytes} [Sun Sep 25 04:23:44 2022] GET /identity/v3/services/object-store => generated 93 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:44.648124 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adb51c28-04b9-4dac-bc2e-a612c8dae236 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:44.720323 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c6990ef-2ec2-4845-9265-676ffca10b05 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:44.720783 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c6990ef-2ec2-4845-9265-676ffca10b05 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:44.721059 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c6990ef-2ec2-4845-9265-676ffca10b05 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:44.723555 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 469/948] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:23:44 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:23:44.729756 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-213a2177-2fb5-401c-8504-b1560c53c59f None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:44.730458 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-213a2177-2fb5-401c-8504-b1560c53c59f None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:44.730835 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-213a2177-2fb5-401c-8504-b1560c53c59f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:44.755859 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adb51c28-04b9-4dac-bc2e-a612c8dae236 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:44.756605 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-213a2177-2fb5-401c-8504-b1560c53c59f None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:23:44.759126 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-213a2177-2fb5-401c-8504-b1560c53c59f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:23:44.800262 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7d56ab7-f9ef-40c8-8ac7-0549c8dd77ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:44.801132 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7d56ab7-f9ef-40c8-8ac7-0549c8dd77ba None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:44.801512 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7d56ab7-f9ef-40c8-8ac7-0549c8dd77ba None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:44.801876 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7d56ab7-f9ef-40c8-8ac7-0549c8dd77ba None admin] PATH_INFO: `/v3/services` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:44.802699 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7d56ab7-f9ef-40c8-8ac7-0549c8dd77ba None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:44.803007 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7d56ab7-f9ef-40c8-8ac7-0549c8dd77ba None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:44.804615 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7d56ab7-f9ef-40c8-8ac7-0549c8dd77ba None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:44.815818 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 480/949] 10.222.0.25 () {56 vars in 1122 bytes} [Sun Sep 25 04:23:44 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:44.816934 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7d56ab7-f9ef-40c8-8ac7-0549c8dd77ba None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:44.836488 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 470/950] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:23:44 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 04:23:44.846584 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:44.923585 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfa9de1f-fee2-448a-896e-62c1d5690975 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:44.923916 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfa9de1f-fee2-448a-896e-62c1d5690975 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:44.924133 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfa9de1f-fee2-448a-896e-62c1d5690975 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:44.925731 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 471/951] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:23:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:23:44.930519 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-555c3414-ce81-4138-9a7d-7309c218e92d None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:44.930801 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-555c3414-ce81-4138-9a7d-7309c218e92d None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:44.931073 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-555c3414-ce81-4138-9a7d-7309c218e92d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:44.936941 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7d56ab7-f9ef-40c8-8ac7-0549c8dd77ba None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:44.938781 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:44.951223 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-555c3414-ce81-4138-9a7d-7309c218e92d None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:23:44.952702 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-555c3414-ce81-4138-9a7d-7309c218e92d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:23:44.974861 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e98830ae-a863-412f-8027-200c9c166661 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:44.975703 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e98830ae-a863-412f-8027-200c9c166661 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:44.976123 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e98830ae-a863-412f-8027-200c9c166661 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:44.976541 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e98830ae-a863-412f-8027-200c9c166661 None admin] PATH_INFO: `/v3/services` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:44.977329 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e98830ae-a863-412f-8027-200c9c166661 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:44.977774 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e98830ae-a863-412f-8027-200c9c166661 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:44.979275 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e98830ae-a863-412f-8027-200c9c166661 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:44.985631 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 481/952] 10.222.0.25 () {56 vars in 1122 bytes} [Sun Sep 25 04:23:44 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:44.990772 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e98830ae-a863-412f-8027-200c9c166661 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:45.043806 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 472/953] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:23:44 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 25 04:23:45.091606 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:45.117445 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e98830ae-a863-412f-8027-200c9c166661 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:45.139901 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-555c3414-ce81-4138-9a7d-7309c218e92d None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:45.155839 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3f79dc6-d158-4dce-bbc0-69f4c4f392a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:45.157070 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3f79dc6-d158-4dce-bbc0-69f4c4f392a6 None admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:45.157566 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3f79dc6-d158-4dce-bbc0-69f4c4f392a6 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:45.158031 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3f79dc6-d158-4dce-bbc0-69f4c4f392a6 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:45.158910 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3f79dc6-d158-4dce-bbc0-69f4c4f392a6 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:45.160623 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3f79dc6-d158-4dce-bbc0-69f4c4f392a6 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:45.183205 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 482/954] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:23:44 2022] POST /identity/v3/endpoints => generated 349 bytes in 193 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:23:45.255343 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-555c3414-ce81-4138-9a7d-7309c218e92d None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:45.294882 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:45.301584 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76d6d7fe-5923-4814-b320-c309c0679fb0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:45.302262 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76d6d7fe-5923-4814-b320-c309c0679fb0 None admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:45.302906 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76d6d7fe-5923-4814-b320-c309c0679fb0 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:45.303180 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76d6d7fe-5923-4814-b320-c309c0679fb0 None admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:45.303834 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76d6d7fe-5923-4814-b320-c309c0679fb0 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:45.305330 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76d6d7fe-5923-4814-b320-c309c0679fb0 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:45.307986 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: WARNING keystone.server.flask.common [None req-76d6d7fe-5923-4814-b320-c309c0679fb0 None admin] Deprecated: Not specifying a domain during a create user, group or project call, and relying on falling back to the default domain, is deprecated as of Liberty. There is no plan to remove this compatibility, however, future API versions may remove this, so please specify the domain explicitly or use a domain-scoped token. Sep 25 04:23:45.329729 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.sql.core [None req-76d6d7fe-5923-4814-b320-c309c0679fb0 None admin] Conflict project: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Sep 25 04:23:45.329729 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Sep 25 04:23:45.329729 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [parameters: {'id': '2e440804e618490dac0077137bbcae48', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Sep 25 04:23:45.329729 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81433) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Sep 25 04:23:45.349146 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-76d6d7fe-5923-4814-b320-c309c0679fb0 None admin] Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 2e440804e618490dac0077137bbcae48.: keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 2e440804e618490dac0077137bbcae48. Sep 25 04:23:45.349146 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:45.349146 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 25 04:23:45.349146 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 25 04:23:45.349146 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 25 04:23:45.349146 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 25 04:23:45.349146 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 25 04:23:45.349146 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application result = self._query(query) Sep 25 04:23:45.349146 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 25 04:23:45.349146 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application conn.query(q) Sep 25 04:23:45.349146 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 25 04:23:45.349146 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 25 04:23:45.349146 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 25 04:23:45.350050 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application result.read() Sep 25 04:23:45.350050 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 25 04:23:45.350050 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 25 04:23:45.350050 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 25 04:23:45.350050 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 25 04:23:45.350050 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 25 04:23:45.350050 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 25 04:23:45.350050 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 25 04:23:45.350050 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 25 04:23:45.350050 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Sep 25 04:23:45.350050 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:45.350050 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Sep 25 04:23:45.350050 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:45.350050 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:45.350050 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Sep 25 04:23:45.350050 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return method(*args, **kwargs) Sep 25 04:23:45.350050 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 241, in create_project Sep 25 04:23:45.350050 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return project_ref.to_dict() Sep 25 04:23:45.351041 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 25 04:23:45.351041 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application next(self.gen) Sep 25 04:23:45.351041 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Sep 25 04:23:45.351041 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application yield resource Sep 25 04:23:45.351041 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 25 04:23:45.351041 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application next(self.gen) Sep 25 04:23:45.351041 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Sep 25 04:23:45.351041 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self.session.rollback() Sep 25 04:23:45.351041 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Sep 25 04:23:45.351041 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self.force_reraise() Sep 25 04:23:45.351041 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Sep 25 04:23:45.351041 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise self.value Sep 25 04:23:45.351041 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Sep 25 04:23:45.351041 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Sep 25 04:23:45.351041 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Sep 25 04:23:45.351041 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application session.commit() Sep 25 04:23:45.351041 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Sep 25 04:23:45.351965 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Sep 25 04:23:45.351965 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Sep 25 04:23:45.351965 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self._prepare_impl() Sep 25 04:23:45.351965 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Sep 25 04:23:45.351965 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self.session.flush() Sep 25 04:23:45.351965 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Sep 25 04:23:45.351965 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self._flush(objects) Sep 25 04:23:45.351965 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Sep 25 04:23:45.351965 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Sep 25 04:23:45.351965 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Sep 25 04:23:45.351965 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application compat.raise_( Sep 25 04:23:45.351965 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 25 04:23:45.351965 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception Sep 25 04:23:45.351965 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Sep 25 04:23:45.351965 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application flush_context.execute() Sep 25 04:23:45.351965 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Sep 25 04:23:45.351965 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rec.execute(self) Sep 25 04:23:45.352946 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Sep 25 04:23:45.352946 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Sep 25 04:23:45.352946 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Sep 25 04:23:45.352946 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application _emit_insert_statements( Sep 25 04:23:45.352946 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Sep 25 04:23:45.352946 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application c = connection._execute_20( Sep 25 04:23:45.352946 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Sep 25 04:23:45.352946 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Sep 25 04:23:45.352946 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Sep 25 04:23:45.352946 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Sep 25 04:23:45.352946 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Sep 25 04:23:45.352946 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ret = self._execute_context( Sep 25 04:23:45.352946 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Sep 25 04:23:45.352946 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Sep 25 04:23:45.352946 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Sep 25 04:23:45.353772 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Sep 25 04:23:45.353772 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 25 04:23:45.353772 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception Sep 25 04:23:45.353772 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 25 04:23:45.353772 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 25 04:23:45.353772 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 25 04:23:45.353772 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 25 04:23:45.353772 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 25 04:23:45.353772 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application result = self._query(query) Sep 25 04:23:45.353772 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 25 04:23:45.353772 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application conn.query(q) Sep 25 04:23:45.353772 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 25 04:23:45.353772 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 25 04:23:45.353772 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 25 04:23:45.353772 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application result.read() Sep 25 04:23:45.353772 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 25 04:23:45.353772 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 25 04:23:45.354676 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 25 04:23:45.354676 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 25 04:23:45.354676 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 25 04:23:45.354676 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 25 04:23:45.354676 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 25 04:23:45.354676 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 25 04:23:45.354676 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Sep 25 04:23:45.354676 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Sep 25 04:23:45.354676 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [parameters: {'id': '2e440804e618490dac0077137bbcae48', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Sep 25 04:23:45.354676 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Sep 25 04:23:45.354676 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:45.354676 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 25 04:23:45.354676 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:45.354676 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:45.354676 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 216, in create_project Sep 25 04:23:45.355474 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ret = self.driver.create_project(project_id, project) Sep 25 04:23:45.355474 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Sep 25 04:23:45.355474 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Sep 25 04:23:45.355474 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - Duplicate entry found with name service at domain ID default. Sep 25 04:23:45.355474 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:45.355474 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 25 04:23:45.355474 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:45.355474 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:45.355474 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:45.355474 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:45.355474 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:45.355474 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:45.355474 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:45.355474 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:45.355474 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:45.355474 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:45.355474 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:45.358779 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:45.358779 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 176, in post Sep 25 04:23:45.358779 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = PROVIDERS.resource_api.create_project( Sep 25 04:23:45.358779 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:45.358779 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:45.358779 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 218, in create_project Sep 25 04:23:45.358779 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.Conflict( Sep 25 04:23:45.358779 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 2e440804e618490dac0077137bbcae48. Sep 25 04:23:45.358779 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:45.358779 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 473/955] 10.222.0.25 () {60 vars in 1140 bytes} [Sun Sep 25 04:23:45 2022] POST /identity/v3/projects => generated 262 bytes in 212 msecs (HTTP/1.1 409) 5 headers in 183 bytes (2 switches on core 0) Sep 25 04:23:45.361536 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3f79dc6-d158-4dce-bbc0-69f4c4f392a6 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:45.361690 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76d6d7fe-5923-4814-b320-c309c0679fb0 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:45.484245 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3f79dc6-d158-4dce-bbc0-69f4c4f392a6 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:45.500813 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76d6d7fe-5923-4814-b320-c309c0679fb0 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:45.523786 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af1357e5-9187-4efe-84fa-8c3634ddbd65 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:45.524420 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af1357e5-9187-4efe-84fa-8c3634ddbd65 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:45.524730 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af1357e5-9187-4efe-84fa-8c3634ddbd65 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:45.525003 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af1357e5-9187-4efe-84fa-8c3634ddbd65 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:45.528675 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:45.541193 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af1357e5-9187-4efe-84fa-8c3634ddbd65 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 04:23:45.548798 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-af1357e5-9187-4efe-84fa-8c3634ddbd65 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 25 04:23:45.548798 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:45.548798 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:45.548798 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:45.548798 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:45.548798 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:45.548798 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:45.548798 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:45.548798 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:45.548798 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:45.548798 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:45.548798 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:45.548798 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 04:23:45.548798 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 04:23:45.548798 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 04:23:45.548798 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 04:23:45.549798 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 04:23:45.549798 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 04:23:45.549798 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 04:23:45.549798 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 04:23:45.549798 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:45.549798 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:45.549798 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:45.549798 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:45.549798 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:45.549798 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:45.549798 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:45.549798 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:45.549798 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:45.549798 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:45.549798 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:45.549798 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:45.550704 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:45.550704 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:45.550704 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:45.550704 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:45.550704 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 04:23:45.550704 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 04:23:45.550704 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 04:23:45.550704 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 04:23:45.550704 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 04:23:45.550704 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 04:23:45.550704 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 25 04:23:45.550704 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:23:45.550704 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 483/956] 10.222.0.25 () {56 vars in 1120 bytes} [Sun Sep 25 04:23:45 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:45.552004 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e21532f-e50b-476e-bb75-5a26b6fd84a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:45.552884 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e21532f-e50b-476e-bb75-5a26b6fd84a7 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:45.553338 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e21532f-e50b-476e-bb75-5a26b6fd84a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:45.553772 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e21532f-e50b-476e-bb75-5a26b6fd84a7 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:45.555928 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af1357e5-9187-4efe-84fa-8c3634ddbd65 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:45.567413 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e21532f-e50b-476e-bb75-5a26b6fd84a7 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:45.570415 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e21532f-e50b-476e-bb75-5a26b6fd84a7 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:45.589491 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-2e21532f-e50b-476e-bb75-5a26b6fd84a7 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 04:23:45.589491 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:45.589491 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:45.589491 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:45.589491 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:45.589491 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:45.589491 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:45.589491 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:45.589491 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:45.589491 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:45.589491 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:45.589491 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:45.589491 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 25 04:23:45.589491 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 25 04:23:45.589491 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 25 04:23:45.590504 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 25 04:23:45.590504 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:45.590504 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:45.590504 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:45.590504 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:45.590504 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:45.590504 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:45.590504 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:45.590504 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:45.590504 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:45.590504 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:45.590504 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:45.590504 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:45.590504 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:45.590504 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:45.590504 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:45.590504 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:45.591417 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 25 04:23:45.591417 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 25 04:23:45.591417 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 25 04:23:45.591417 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 25 04:23:45.591417 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 25 04:23:45.591417 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 25 04:23:45.591417 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 04:23:45.591417 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:45.594792 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 474/957] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Sep 25 04:23:45 2022] GET /identity/v3/projects/service => generated 88 bytes in 234 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:45.601327 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e21532f-e50b-476e-bb75-5a26b6fd84a7 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:45.662233 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:45.685573 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af1357e5-9187-4efe-84fa-8c3634ddbd65 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:45.723433 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32cf38a9-04e8-4e07-a4a5-ce7944577845 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:45.724051 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32cf38a9-04e8-4e07-a4a5-ce7944577845 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:45.724344 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32cf38a9-04e8-4e07-a4a5-ce7944577845 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:45.724607 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32cf38a9-04e8-4e07-a4a5-ce7944577845 None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:45.725316 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32cf38a9-04e8-4e07-a4a5-ce7944577845 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:45.725624 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32cf38a9-04e8-4e07-a4a5-ce7944577845 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:45.726996 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32cf38a9-04e8-4e07-a4a5-ce7944577845 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:45.731252 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e21532f-e50b-476e-bb75-5a26b6fd84a7 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:45.736232 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 484/958] 10.222.0.25 () {56 vars in 1115 bytes} [Sun Sep 25 04:23:45 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:45.737476 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-937c5084-1463-434d-a2c7-44cad561ff37 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:45.737763 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-937c5084-1463-434d-a2c7-44cad561ff37 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:45.738035 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-937c5084-1463-434d-a2c7-44cad561ff37 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:45.740769 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 485/959] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:23:45 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:23:45.743892 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-937c5084-1463-434d-a2c7-44cad561ff37 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:45.772034 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5775ff3-bf88-4c95-b2a9-ccb6ee0b4dab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:45.772668 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5775ff3-bf88-4c95-b2a9-ccb6ee0b4dab None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:45.773050 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5775ff3-bf88-4c95-b2a9-ccb6ee0b4dab None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:45.773332 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5775ff3-bf88-4c95-b2a9-ccb6ee0b4dab None admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:45.774058 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5775ff3-bf88-4c95-b2a9-ccb6ee0b4dab None admin] RBAC: Adding query filter params (name=service) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:45.774312 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5775ff3-bf88-4c95-b2a9-ccb6ee0b4dab None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:45.775821 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5775ff3-bf88-4c95-b2a9-ccb6ee0b4dab None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:45.796470 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 475/960] 10.222.0.25 () {56 vars in 1112 bytes} [Sun Sep 25 04:23:45 2022] GET /identity/v3/projects?name=service => generated 411 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:45.797635 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b494bb-2222-48be-9b4a-77802b83e4df None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:45.797929 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b494bb-2222-48be-9b4a-77802b83e4df None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:45.798208 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b494bb-2222-48be-9b4a-77802b83e4df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:45.819167 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-89b494bb-2222-48be-9b4a-77802b83e4df None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:23:45.820789 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-89b494bb-2222-48be-9b4a-77802b83e4df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:23:45.825413 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:45.897211 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-937c5084-1463-434d-a2c7-44cad561ff37 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:45.899069 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 476/961] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:23:45 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 04:23:45.938669 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1db2a592-998f-4336-8ab6-a3a81e5b9be1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:45.939726 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1db2a592-998f-4336-8ab6-a3a81e5b9be1 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:45.940217 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1db2a592-998f-4336-8ab6-a3a81e5b9be1 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:45.940729 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1db2a592-998f-4336-8ab6-a3a81e5b9be1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:45.953878 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1db2a592-998f-4336-8ab6-a3a81e5b9be1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:45.955797 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1db2a592-998f-4336-8ab6-a3a81e5b9be1 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:45.972463 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-1db2a592-998f-4336-8ab6-a3a81e5b9be1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:23:45.972463 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:45.972463 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:45.972463 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:45.972463 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:45.972463 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:45.972463 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:45.972463 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:45.972463 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:45.972463 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:45.972463 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:45.972463 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:45.972463 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 04:23:45.972463 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 04:23:45.972463 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 04:23:45.972463 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 04:23:45.973758 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:45.973758 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:45.973758 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:45.973758 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:45.973758 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:45.973758 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:45.973758 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:45.973758 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:45.973758 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:45.973758 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:45.973758 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:45.973758 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:45.973758 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:45.973758 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:45.973758 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:45.973758 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:45.973758 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 04:23:45.974910 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 04:23:45.974910 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:23:45.974910 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:23:45.976715 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 486/962] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:23:45 2022] GET /identity/v3/domains/Default => generated 87 bytes in 233 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:45.983261 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89b494bb-2222-48be-9b4a-77802b83e4df None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:46.023624 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:46.098174 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89b494bb-2222-48be-9b4a-77802b83e4df None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:46.134011 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-804a6ca2-1a96-41c3-be21-96f44f0a2ddc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:46.134630 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-804a6ca2-1a96-41c3-be21-96f44f0a2ddc None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:46.134913 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-804a6ca2-1a96-41c3-be21-96f44f0a2ddc None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:46.135187 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-804a6ca2-1a96-41c3-be21-96f44f0a2ddc None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:46.135841 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-804a6ca2-1a96-41c3-be21-96f44f0a2ddc None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:46.136135 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-804a6ca2-1a96-41c3-be21-96f44f0a2ddc None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:46.137598 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-804a6ca2-1a96-41c3-be21-96f44f0a2ddc None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:46.150094 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 477/963] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Sep 25 04:23:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:46.162150 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1db2a592-998f-4336-8ab6-a3a81e5b9be1 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:46.180309 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-804a6ca2-1a96-41c3-be21-96f44f0a2ddc None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:46.269513 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1db2a592-998f-4336-8ab6-a3a81e5b9be1 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:46.282755 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:46.285087 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:46.292917 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-804a6ca2-1a96-41c3-be21-96f44f0a2ddc None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:46.307666 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-779d35c9-f032-4f38-b93f-1121867cc91b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:46.308785 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-779d35c9-f032-4f38-b93f-1121867cc91b None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:46.309634 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-779d35c9-f032-4f38-b93f-1121867cc91b None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:46.310230 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-779d35c9-f032-4f38-b93f-1121867cc91b None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:46.319487 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-779d35c9-f032-4f38-b93f-1121867cc91b None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:46.321826 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-779d35c9-f032-4f38-b93f-1121867cc91b None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:46.332181 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f5e9a47-eae9-4778-bc29-e68babaf6f05 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:46.332954 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f5e9a47-eae9-4778-bc29-e68babaf6f05 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:46.333353 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f5e9a47-eae9-4778-bc29-e68babaf6f05 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:46.333704 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f5e9a47-eae9-4778-bc29-e68babaf6f05 None admin] PATH_INFO: `/v3/services/compute` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:46.334366 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f5e9a47-eae9-4778-bc29-e68babaf6f05 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:46.335804 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f5e9a47-eae9-4778-bc29-e68babaf6f05 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:46.336357 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-779d35c9-f032-4f38-b93f-1121867cc91b None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 25 04:23:46.336357 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:46.336357 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:46.336357 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:46.336357 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:46.336357 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:46.336357 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:46.336357 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:46.336357 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:46.336357 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:46.336357 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:46.336357 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:46.336357 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 25 04:23:46.336357 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 25 04:23:46.336357 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 25 04:23:46.336357 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 25 04:23:46.337601 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:46.337601 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:46.337601 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 25 04:23:46.337601 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 04:23:46.337601 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 25 04:23:46.337601 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 04:23:46.337601 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:46.337601 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:46.337601 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:46.337601 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:46.337601 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:46.337601 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:46.337601 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:46.337601 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:46.337601 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:46.337601 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:46.337601 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:46.337601 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:46.338981 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:46.338981 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:46.338981 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 25 04:23:46.338981 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 25 04:23:46.338981 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 25 04:23:46.338981 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 25 04:23:46.338981 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 25 04:23:46.338981 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 25 04:23:46.338981 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 25 04:23:46.338981 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:23:46.341174 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 487/964] 10.222.0.25 () {56 vars in 1117 bytes} [Sun Sep 25 04:23:46 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:46.352908 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-779d35c9-f032-4f38-b93f-1121867cc91b None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:46.353074 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-8f5e9a47-eae9-4778-bc29-e68babaf6f05 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Sep 25 04:23:46.353074 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:46.353074 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 25 04:23:46.353074 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 25 04:23:46.353074 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 25 04:23:46.353074 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 25 04:23:46.353074 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 25 04:23:46.353074 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 04:23:46.353074 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 25 04:23:46.353074 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:46.353074 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 25 04:23:46.353074 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:46.353074 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:46.353074 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:46.353074 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:46.353074 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:46.353074 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:46.354205 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:46.354205 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:46.354205 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:46.354205 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:46.354205 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:46.354205 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:46.354205 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 25 04:23:46.354205 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 25 04:23:46.354205 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 25 04:23:46.354205 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 25 04:23:46.354205 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:46.354205 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:46.354205 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:46.354205 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:46.354205 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:46.354205 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:46.355239 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:46.355239 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:46.355239 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:46.355239 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:46.355239 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:46.355239 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:46.355239 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:46.355239 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:46.355239 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:46.355239 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:46.355239 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 25 04:23:46.355239 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 04:23:46.355239 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 25 04:23:46.355239 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:46.355239 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 478/965] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Sep 25 04:23:46 2022] GET /identity/v3/services/compute => generated 88 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:46.361715 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f5e9a47-eae9-4778-bc29-e68babaf6f05 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:46.459777 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-779d35c9-f032-4f38-b93f-1121867cc91b None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:46.470982 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f5e9a47-eae9-4778-bc29-e68babaf6f05 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:46.503535 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d915212-2bde-4f69-b1ac-b319a398aee6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:46.504482 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d915212-2bde-4f69-b1ac-b319a398aee6 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:46.504945 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d915212-2bde-4f69-b1ac-b319a398aee6 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:46.505418 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d915212-2bde-4f69-b1ac-b319a398aee6 None admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:46.506351 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d915212-2bde-4f69-b1ac-b319a398aee6 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:46.506847 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d915212-2bde-4f69-b1ac-b319a398aee6 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:46.508455 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d915212-2bde-4f69-b1ac-b319a398aee6 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:46.509121 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af29b676-858b-4aa1-ac22-c198b5c27e59 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:46.509795 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af29b676-858b-4aa1-ac22-c198b5c27e59 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:46.510139 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af29b676-858b-4aa1-ac22-c198b5c27e59 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:46.510479 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af29b676-858b-4aa1-ac22-c198b5c27e59 None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:46.511359 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af29b676-858b-4aa1-ac22-c198b5c27e59 None admin] RBAC: Adding query filter params (name=compute) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:46.511753 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af29b676-858b-4aa1-ac22-c198b5c27e59 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:46.513999 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af29b676-858b-4aa1-ac22-c198b5c27e59 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:46.524505 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 479/966] 10.222.0.25 () {56 vars in 1112 bytes} [Sun Sep 25 04:23:46 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:46.526129 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af29b676-858b-4aa1-ac22-c198b5c27e59 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:46.535719 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 488/967] 10.222.0.25 () {56 vars in 1149 bytes} [Sun Sep 25 04:23:46 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:23:46.537430 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d915212-2bde-4f69-b1ac-b319a398aee6 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:46.554660 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:46.577991 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:46.640776 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af29b676-858b-4aa1-ac22-c198b5c27e59 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:46.662249 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d915212-2bde-4f69-b1ac-b319a398aee6 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:46.677599 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eca55987-fb48-4ea7-9953-f89665f5c562 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:46.678346 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eca55987-fb48-4ea7-9953-f89665f5c562 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:46.678658 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eca55987-fb48-4ea7-9953-f89665f5c562 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:46.678998 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eca55987-fb48-4ea7-9953-f89665f5c562 None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:46.679699 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eca55987-fb48-4ea7-9953-f89665f5c562 None admin] RBAC: Adding query filter params (type=compute) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:46.680289 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eca55987-fb48-4ea7-9953-f89665f5c562 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:46.681550 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eca55987-fb48-4ea7-9953-f89665f5c562 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:46.692680 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 480/968] 10.222.0.25 () {56 vars in 1112 bytes} [Sun Sep 25 04:23:46 2022] GET /identity/v3/services?type=compute => generated 124 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:23:46.708973 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97247dc7-4f30-4a74-9b13-31ca98beab91 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:46.709681 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97247dc7-4f30-4a74-9b13-31ca98beab91 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:46.709928 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97247dc7-4f30-4a74-9b13-31ca98beab91 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:46.710206 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97247dc7-4f30-4a74-9b13-31ca98beab91 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:46.787257 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97247dc7-4f30-4a74-9b13-31ca98beab91 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:46.788757 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97247dc7-4f30-4a74-9b13-31ca98beab91 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:46.818854 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:46.902322 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 489/969] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Sep 25 04:23:46 2022] GET /identity/v3/auth/tokens => generated 1474 bytes in 365 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Sep 25 04:23:46.909960 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eca55987-fb48-4ea7-9953-f89665f5c562 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:47.010976 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:47.023564 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eca55987-fb48-4ea7-9953-f89665f5c562 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:47.059205 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39451513-a761-4fc4-927e-85c99613e6e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:47.059889 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39451513-a761-4fc4-927e-85c99613e6e8 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:47.060173 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39451513-a761-4fc4-927e-85c99613e6e8 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:47.060442 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39451513-a761-4fc4-927e-85c99613e6e8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:47.070613 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39451513-a761-4fc4-927e-85c99613e6e8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:47.072051 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39451513-a761-4fc4-927e-85c99613e6e8 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:47.087636 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-39451513-a761-4fc4-927e-85c99613e6e8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:23:47.087636 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:47.087636 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:47.087636 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:47.087636 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:47.087636 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:47.087636 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:47.087636 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:47.087636 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:47.087636 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:47.087636 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:47.087636 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:47.087636 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 04:23:47.087636 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 04:23:47.087636 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 04:23:47.087636 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 04:23:47.088744 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:47.088744 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:47.088744 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:47.088744 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:47.088744 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:47.088744 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:47.088744 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:47.088744 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:47.088744 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:47.088744 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:47.088744 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:47.088744 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:47.088744 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:47.088744 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:47.088744 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:47.088744 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:47.088744 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 04:23:47.089710 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 04:23:47.089710 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:23:47.089710 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:47.089710 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 481/970] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:23:46 2022] GET /identity/v3/domains/Default => generated 87 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:47.095701 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97247dc7-4f30-4a74-9b13-31ca98beab91 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:47.217994 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97247dc7-4f30-4a74-9b13-31ca98beab91 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:47.245271 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:47.254014 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c53caed-9dc3-4db0-b70e-455b9cd62a1a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:47.254633 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c53caed-9dc3-4db0-b70e-455b9cd62a1a None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:47.254913 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c53caed-9dc3-4db0-b70e-455b9cd62a1a None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:47.255182 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c53caed-9dc3-4db0-b70e-455b9cd62a1a None admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:47.255824 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c53caed-9dc3-4db0-b70e-455b9cd62a1a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:47.256130 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c53caed-9dc3-4db0-b70e-455b9cd62a1a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:47.257524 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c53caed-9dc3-4db0-b70e-455b9cd62a1a None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:47.274737 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 490/971] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Sep 25 04:23:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:23:47.280922 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39451513-a761-4fc4-927e-85c99613e6e8 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:47.386658 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39451513-a761-4fc4-927e-85c99613e6e8 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:47.423626 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d1ff6b4-7025-4abe-a11a-66c6e3084fa4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:47.424705 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d1ff6b4-7025-4abe-a11a-66c6e3084fa4 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:47.425212 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d1ff6b4-7025-4abe-a11a-66c6e3084fa4 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:47.425653 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d1ff6b4-7025-4abe-a11a-66c6e3084fa4 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:47.432945 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d1ff6b4-7025-4abe-a11a-66c6e3084fa4 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:47.434382 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d1ff6b4-7025-4abe-a11a-66c6e3084fa4 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:47.437342 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:47.446572 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-7d1ff6b4-7025-4abe-a11a-66c6e3084fa4 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 04:23:47.446572 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:47.446572 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:47.446572 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:47.446572 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:47.446572 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:47.446572 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:47.446572 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:47.446572 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:47.446572 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:47.446572 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:47.446572 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:47.446572 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 25 04:23:47.446572 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 25 04:23:47.446572 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 25 04:23:47.447731 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 25 04:23:47.447731 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:47.447731 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:47.447731 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:47.447731 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:47.447731 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:47.447731 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:47.447731 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:47.447731 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:47.447731 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:47.447731 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:47.447731 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:47.447731 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:47.447731 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:47.447731 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:47.447731 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:47.447731 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:47.448926 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 25 04:23:47.448926 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 25 04:23:47.448926 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 25 04:23:47.448926 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 25 04:23:47.448926 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 25 04:23:47.448926 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 25 04:23:47.448926 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 04:23:47.448926 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:47.451657 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 482/972] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Sep 25 04:23:47 2022] GET /identity/v3/projects/service => generated 88 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:47.457443 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c53caed-9dc3-4db0-b70e-455b9cd62a1a None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:47.594246 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c53caed-9dc3-4db0-b70e-455b9cd62a1a None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:47.618378 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13483399-1961-4bc2-8e91-83267e41b4a1 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:47.619027 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13483399-1961-4bc2-8e91-83267e41b4a1 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:47.619540 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13483399-1961-4bc2-8e91-83267e41b4a1 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:47.621377 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 483/973] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:23:47 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:23:47.627258 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0207eda-e61e-4907-ba2b-1dedf7a61e5f None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:47.627785 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0207eda-e61e-4907-ba2b-1dedf7a61e5f None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:47.628258 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0207eda-e61e-4907-ba2b-1dedf7a61e5f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:47.630048 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab8ac9c6-b4c6-4fe7-afbe-d5623e2cb05d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:47.630657 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab8ac9c6-b4c6-4fe7-afbe-d5623e2cb05d None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:47.630931 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab8ac9c6-b4c6-4fe7-afbe-d5623e2cb05d None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:47.631199 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab8ac9c6-b4c6-4fe7-afbe-d5623e2cb05d None admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:47.631886 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab8ac9c6-b4c6-4fe7-afbe-d5623e2cb05d None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:47.632191 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab8ac9c6-b4c6-4fe7-afbe-d5623e2cb05d None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:47.633633 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab8ac9c6-b4c6-4fe7-afbe-d5623e2cb05d None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:47.648899 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-e0207eda-e61e-4907-ba2b-1dedf7a61e5f None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:23:47.649215 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 491/974] 10.222.0.25 () {56 vars in 1148 bytes} [Sun Sep 25 04:23:47 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:47.650592 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-e0207eda-e61e-4907-ba2b-1dedf7a61e5f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:23:47.654876 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab8ac9c6-b4c6-4fe7-afbe-d5623e2cb05d None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:47.711114 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:47.752086 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 484/975] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:23:47 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 25 04:23:47.769831 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab8ac9c6-b4c6-4fe7-afbe-d5623e2cb05d None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:47.806178 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b8ac112-ecca-433c-824f-25cadb6ccb87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:47.806839 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b8ac112-ecca-433c-824f-25cadb6ccb87 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:47.807123 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b8ac112-ecca-433c-824f-25cadb6ccb87 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:47.807397 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b8ac112-ecca-433c-824f-25cadb6ccb87 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:47.808131 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b8ac112-ecca-433c-824f-25cadb6ccb87 None admin] RBAC: Adding query filter params (role.id=062e3575fa1d40c880fc87ceca94db4a, scope.project.id=056ca1b1a9974a7581550de0d9217823, user.id=050b3aab44bb4b17aa4a1ba7cccdb231) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:47.808442 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b8ac112-ecca-433c-824f-25cadb6ccb87 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:47.809899 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b8ac112-ecca-433c-824f-25cadb6ccb87 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:47.816339 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 492/976] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Sep 25 04:23:47 2022] GET /identity/v3/role_assignments?user.id=050b3aab44bb4b17aa4a1ba7cccdb231&scope.project.id=056ca1b1a9974a7581550de0d9217823&role.id=062e3575fa1d40c880fc87ceca94db4a => generated 608 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:23:48.026619 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0207eda-e61e-4907-ba2b-1dedf7a61e5f None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:48.134186 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0207eda-e61e-4907-ba2b-1dedf7a61e5f None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:48.167261 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:48.175274 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dd4d49c-5aea-439e-b08e-8dab889b2501 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:48.176333 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd4d49c-5aea-439e-b08e-8dab889b2501 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:48.176857 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd4d49c-5aea-439e-b08e-8dab889b2501 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:48.177351 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd4d49c-5aea-439e-b08e-8dab889b2501 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:48.178194 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dd4d49c-5aea-439e-b08e-8dab889b2501 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:48.179854 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dd4d49c-5aea-439e-b08e-8dab889b2501 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:48.190704 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-2dd4d49c-5aea-439e-b08e-8dab889b2501 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 25 04:23:48.190704 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:48.190704 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 25 04:23:48.190704 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 25 04:23:48.190704 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 25 04:23:48.190704 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 25 04:23:48.190704 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 25 04:23:48.190704 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 04:23:48.190704 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 25 04:23:48.190704 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:48.190704 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 25 04:23:48.190704 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:48.190704 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:48.190704 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:48.190704 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:48.190704 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:48.190704 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:48.191820 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:48.191820 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:48.191820 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:48.191820 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:48.191820 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:48.191820 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:48.191820 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 25 04:23:48.191820 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 25 04:23:48.191820 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 25 04:23:48.191820 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 25 04:23:48.191820 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:48.191820 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:48.191820 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:48.191820 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:48.191820 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:48.191820 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:48.192716 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:48.192716 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:48.192716 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:48.192716 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:48.192716 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:48.192716 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:48.192716 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:48.192716 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:48.192716 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:48.192716 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:48.192716 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 25 04:23:48.192716 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 04:23:48.192716 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 25 04:23:48.192716 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:48.195714 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 485/977] 10.222.0.25 () {56 vars in 1126 bytes} [Sun Sep 25 04:23:48 2022] GET /identity/v3/services/object-store => generated 93 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:48.202064 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b8ac112-ecca-433c-824f-25cadb6ccb87 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:48.302743 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2ab960a-a2d8-4e14-bc33-e1dd3ad1084f None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:48.303674 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2ab960a-a2d8-4e14-bc33-e1dd3ad1084f None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:48.304160 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2ab960a-a2d8-4e14-bc33-e1dd3ad1084f None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:48.306011 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 486/978] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:23:48 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 25 04:23:48.308196 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b8ac112-ecca-433c-824f-25cadb6ccb87 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:48.311003 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e66b3123-82d7-4594-ab81-fcf9f564ab4d None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:48.311726 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e66b3123-82d7-4594-ab81-fcf9f564ab4d None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:48.312355 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e66b3123-82d7-4594-ab81-fcf9f564ab4d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:48.322332 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:48.333506 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-e66b3123-82d7-4594-ab81-fcf9f564ab4d None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:23:48.335130 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-e66b3123-82d7-4594-ab81-fcf9f564ab4d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:23:48.347565 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-896717ab-e332-4a23-8b58-735a9f690b08 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:48.348182 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-896717ab-e332-4a23-8b58-735a9f690b08 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:48.348463 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-896717ab-e332-4a23-8b58-735a9f690b08 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:48.348762 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-896717ab-e332-4a23-8b58-735a9f690b08 None admin] PATH_INFO: `/v3/services` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:48.349460 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-896717ab-e332-4a23-8b58-735a9f690b08 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:48.349686 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-896717ab-e332-4a23-8b58-735a9f690b08 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:48.351122 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-896717ab-e332-4a23-8b58-735a9f690b08 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:48.357867 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 493/979] 10.222.0.25 () {56 vars in 1122 bytes} [Sun Sep 25 04:23:48 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:48.364264 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-896717ab-e332-4a23-8b58-735a9f690b08 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:48.416902 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 487/980] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:23:48 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 25 04:23:48.475477 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-896717ab-e332-4a23-8b58-735a9f690b08 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:48.510255 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d199f603-efdc-41c4-b118-6966324042a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:48.510876 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d199f603-efdc-41c4-b118-6966324042a1 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:48.511156 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d199f603-efdc-41c4-b118-6966324042a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:48.511430 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d199f603-efdc-41c4-b118-6966324042a1 None admin] PATH_INFO: `/v3/services` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:48.512063 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d199f603-efdc-41c4-b118-6966324042a1 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:48.512368 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d199f603-efdc-41c4-b118-6966324042a1 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:48.513775 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d199f603-efdc-41c4-b118-6966324042a1 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:48.519330 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 494/981] 10.222.0.25 () {56 vars in 1122 bytes} [Sun Sep 25 04:23:48 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:23:48.524834 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e66b3123-82d7-4594-ab81-fcf9f564ab4d None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:48.634822 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e66b3123-82d7-4594-ab81-fcf9f564ab4d None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:48.658933 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:48.670830 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fb141c7-1ae3-4c81-9541-ac3370da80e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:48.671795 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fb141c7-1ae3-4c81-9541-ac3370da80e2 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:48.672107 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fb141c7-1ae3-4c81-9541-ac3370da80e2 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:48.672378 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fb141c7-1ae3-4c81-9541-ac3370da80e2 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:48.673265 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fb141c7-1ae3-4c81-9541-ac3370da80e2 None admin] RBAC: Adding query filter params (interface=admin, service_id=fae12ee1be974fce89b644eeb639d0c4, region_id=RegionOne) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:48.673583 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fb141c7-1ae3-4c81-9541-ac3370da80e2 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:48.674958 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fb141c7-1ae3-4c81-9541-ac3370da80e2 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:48.680607 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 488/982] 10.222.0.25 () {56 vars in 1249 bytes} [Sun Sep 25 04:23:48 2022] GET /identity/v3/endpoints?service_id=fae12ee1be974fce89b644eeb639d0c4&interface=admin®ion_id=RegionOne => generated 193 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:48.686282 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d199f603-efdc-41c4-b118-6966324042a1 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:48.699376 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fb141c7-1ae3-4c81-9541-ac3370da80e2 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:48.741744 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:48.804934 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fb141c7-1ae3-4c81-9541-ac3370da80e2 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:48.824266 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d199f603-efdc-41c4-b118-6966324042a1 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:48.840491 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce908708-f5a6-4d53-9340-39129464cf1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:48.841218 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce908708-f5a6-4d53-9340-39129464cf1b None admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:48.841565 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce908708-f5a6-4d53-9340-39129464cf1b None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:48.841901 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce908708-f5a6-4d53-9340-39129464cf1b None admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:48.842832 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce908708-f5a6-4d53-9340-39129464cf1b None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:48.844396 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce908708-f5a6-4d53-9340-39129464cf1b None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:48.863290 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.sql.core [None req-ce908708-f5a6-4d53-9340-39129464cf1b None admin] Conflict user: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Sep 25 04:23:48.863290 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Sep 25 04:23:48.863290 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [parameters: {'user_id': '38de5e16653c4b659e6ed0e105d6098d', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Sep 25 04:23:48.863290 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81433) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Sep 25 04:23:48.867696 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6dc857c-605a-4cff-82a5-c5ce00730a25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:48.868521 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6dc857c-605a-4cff-82a5-c5ce00730a25 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:48.869141 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6dc857c-605a-4cff-82a5-c5ce00730a25 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:48.869589 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6dc857c-605a-4cff-82a5-c5ce00730a25 None admin] PATH_INFO: `/v3/services` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:48.870503 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6dc857c-605a-4cff-82a5-c5ce00730a25 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:48.871675 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-ce908708-f5a6-4d53-9340-39129464cf1b None admin] Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default.: keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Sep 25 04:23:48.871675 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:48.871675 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 25 04:23:48.871675 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 25 04:23:48.871675 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 25 04:23:48.871675 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 25 04:23:48.871675 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 25 04:23:48.871675 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application result = self._query(query) Sep 25 04:23:48.871675 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 25 04:23:48.871675 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application conn.query(q) Sep 25 04:23:48.871675 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 25 04:23:48.871675 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 25 04:23:48.871675 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 25 04:23:48.871675 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application result.read() Sep 25 04:23:48.871675 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 25 04:23:48.872577 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 25 04:23:48.872577 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 25 04:23:48.872577 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 25 04:23:48.872577 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 25 04:23:48.872577 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 25 04:23:48.872577 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 25 04:23:48.872577 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 25 04:23:48.872577 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Sep 25 04:23:48.872577 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:48.872577 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Sep 25 04:23:48.872577 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:48.872577 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:48.872577 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Sep 25 04:23:48.872577 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return method(*args, **kwargs) Sep 25 04:23:48.872577 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 141, in create_user Sep 25 04:23:48.872577 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return base.filter_user(user_ref.to_dict()) Sep 25 04:23:48.872577 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 25 04:23:48.872577 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application next(self.gen) Sep 25 04:23:48.873798 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Sep 25 04:23:48.873798 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application yield resource Sep 25 04:23:48.873798 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 25 04:23:48.873798 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application next(self.gen) Sep 25 04:23:48.873798 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Sep 25 04:23:48.873798 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self.session.rollback() Sep 25 04:23:48.873798 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Sep 25 04:23:48.873798 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self.force_reraise() Sep 25 04:23:48.873798 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Sep 25 04:23:48.873798 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise self.value Sep 25 04:23:48.873798 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Sep 25 04:23:48.873798 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Sep 25 04:23:48.873798 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Sep 25 04:23:48.873798 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application session.commit() Sep 25 04:23:48.873798 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Sep 25 04:23:48.873798 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Sep 25 04:23:48.873798 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Sep 25 04:23:48.875138 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self._prepare_impl() Sep 25 04:23:48.875138 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Sep 25 04:23:48.875138 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self.session.flush() Sep 25 04:23:48.875138 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Sep 25 04:23:48.875138 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self._flush(objects) Sep 25 04:23:48.875138 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Sep 25 04:23:48.875138 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Sep 25 04:23:48.875138 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Sep 25 04:23:48.875138 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application compat.raise_( Sep 25 04:23:48.875138 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 25 04:23:48.875138 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception Sep 25 04:23:48.875138 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Sep 25 04:23:48.875138 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application flush_context.execute() Sep 25 04:23:48.875138 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Sep 25 04:23:48.875138 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rec.execute(self) Sep 25 04:23:48.875138 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Sep 25 04:23:48.875138 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Sep 25 04:23:48.876108 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Sep 25 04:23:48.876108 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application _emit_insert_statements( Sep 25 04:23:48.876108 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1238, in _emit_insert_statements Sep 25 04:23:48.876108 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application result = connection._execute_20( Sep 25 04:23:48.876108 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Sep 25 04:23:48.876108 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Sep 25 04:23:48.876108 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Sep 25 04:23:48.876108 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Sep 25 04:23:48.876108 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Sep 25 04:23:48.876108 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ret = self._execute_context( Sep 25 04:23:48.876108 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Sep 25 04:23:48.876108 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Sep 25 04:23:48.876108 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Sep 25 04:23:48.876108 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Sep 25 04:23:48.876108 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 25 04:23:48.876108 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception Sep 25 04:23:48.877027 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 25 04:23:48.877027 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 25 04:23:48.877027 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 25 04:23:48.877027 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 25 04:23:48.877027 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 25 04:23:48.877027 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application result = self._query(query) Sep 25 04:23:48.877027 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 25 04:23:48.877027 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application conn.query(q) Sep 25 04:23:48.877027 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 25 04:23:48.877027 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 25 04:23:48.877027 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 25 04:23:48.877027 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application result.read() Sep 25 04:23:48.877027 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 25 04:23:48.877027 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 25 04:23:48.877027 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 25 04:23:48.877027 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 25 04:23:48.877890 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 25 04:23:48.877890 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 25 04:23:48.877890 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 25 04:23:48.877890 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 25 04:23:48.877890 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Sep 25 04:23:48.877890 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Sep 25 04:23:48.877890 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [parameters: {'user_id': '38de5e16653c4b659e6ed0e105d6098d', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Sep 25 04:23:48.877890 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Sep 25 04:23:48.877890 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:48.877890 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 25 04:23:48.877890 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:48.877890 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:48.877890 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:48.877890 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:48.877890 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:48.878726 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:48.878726 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:48.878726 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:48.878726 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:48.878726 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:48.878726 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:48.878726 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:48.878726 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 244, in post Sep 25 04:23:48.878726 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.create_user( Sep 25 04:23:48.878726 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:48.878726 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:48.878726 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 25 04:23:48.878726 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 04:23:48.878726 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 25 04:23:48.878726 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 04:23:48.878726 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1000, in create_user Sep 25 04:23:48.878726 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = self._create_user_with_federated_objects(user, driver) Sep 25 04:23:48.880020 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 966, in _create_user_with_federated_objects Sep 25 04:23:48.880020 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application user = driver.create_user(user['id'], user) Sep 25 04:23:48.880020 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Sep 25 04:23:48.880020 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Sep 25 04:23:48.880020 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Sep 25 04:23:48.880020 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:48.880020 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 489/983] 10.222.0.25 () {60 vars in 1131 bytes} [Sun Sep 25 04:23:48 2022] POST /identity/v3/users => generated 160 bytes in 179 msecs (HTTP/1.1 409) 5 headers in 183 bytes (2 switches on core 0) Sep 25 04:23:48.880020 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce908708-f5a6-4d53-9340-39129464cf1b None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:48.882792 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6dc857c-605a-4cff-82a5-c5ce00730a25 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:48.896667 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 495/984] 10.222.0.25 () {56 vars in 1087 bytes} [Sun Sep 25 04:23:48 2022] GET /identity/v3/services => generated 1497 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:23:48.954140 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:49.012295 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce908708-f5a6-4d53-9340-39129464cf1b None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:49.052315 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf8a95a3-d3d6-4ee6-9020-b068cf0251bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:49.053388 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf8a95a3-d3d6-4ee6-9020-b068cf0251bc None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:49.053863 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf8a95a3-d3d6-4ee6-9020-b068cf0251bc None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:49.054364 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf8a95a3-d3d6-4ee6-9020-b068cf0251bc None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:49.064129 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf8a95a3-d3d6-4ee6-9020-b068cf0251bc None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:49.066162 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf8a95a3-d3d6-4ee6-9020-b068cf0251bc None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:49.080417 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-cf8a95a3-d3d6-4ee6-9020-b068cf0251bc None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Sep 25 04:23:49.080417 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:49.080417 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:49.080417 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:49.080417 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:49.080417 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:49.080417 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:49.080417 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:49.080417 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:49.080417 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:49.080417 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:49.080417 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:49.080417 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 25 04:23:49.080417 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 25 04:23:49.080417 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 25 04:23:49.080417 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 25 04:23:49.081492 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:49.081492 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:49.081492 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 25 04:23:49.081492 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 04:23:49.081492 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 25 04:23:49.081492 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 04:23:49.081492 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:49.081492 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:49.081492 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:49.081492 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:49.081492 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:49.081492 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:49.081492 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:49.081492 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:49.081492 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:49.081492 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:49.081492 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:49.081492 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:49.082495 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:49.082495 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:49.082495 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 25 04:23:49.082495 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 25 04:23:49.082495 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 25 04:23:49.082495 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 25 04:23:49.082495 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 25 04:23:49.082495 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 25 04:23:49.082495 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Sep 25 04:23:49.082495 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:49.085086 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 490/985] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Sep 25 04:23:48 2022] GET /identity/v3/users/cinder => generated 84 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:49.094523 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6dc857c-605a-4cff-82a5-c5ce00730a25 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:49.104242 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07062545-e776-4627-bab4-d7531c15323e None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:49.104529 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07062545-e776-4627-bab4-d7531c15323e None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:49.104843 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07062545-e776-4627-bab4-d7531c15323e None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:49.109406 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 491/986] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:23:49 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 25 04:23:49.120265 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0f0067d-b8a8-4a8a-8ba2-aa982f827bd2 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:49.120558 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0f0067d-b8a8-4a8a-8ba2-aa982f827bd2 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:49.120861 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0f0067d-b8a8-4a8a-8ba2-aa982f827bd2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:49.144448 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-c0f0067d-b8a8-4a8a-8ba2-aa982f827bd2 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:23:49.146336 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-c0f0067d-b8a8-4a8a-8ba2-aa982f827bd2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:23:49.213293 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6dc857c-605a-4cff-82a5-c5ce00730a25 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:49.216407 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:49.242727 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 492/987] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:23:49 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 25 04:23:49.256944 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bac0c87f-88ac-4f83-9eb0-b2bce11ea79b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:49.257797 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bac0c87f-88ac-4f83-9eb0-b2bce11ea79b None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:49.258253 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bac0c87f-88ac-4f83-9eb0-b2bce11ea79b None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:49.258721 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bac0c87f-88ac-4f83-9eb0-b2bce11ea79b None admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:49.259587 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bac0c87f-88ac-4f83-9eb0-b2bce11ea79b None admin] RBAC: Adding query filter params (name=cinder) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:49.260087 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bac0c87f-88ac-4f83-9eb0-b2bce11ea79b None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:49.261702 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bac0c87f-88ac-4f83-9eb0-b2bce11ea79b None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:49.302027 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 496/988] 10.222.0.25 () {56 vars in 1101 bytes} [Sun Sep 25 04:23:49 2022] GET /identity/v3/users?name=cinder => generated 354 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:49.515644 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0f0067d-b8a8-4a8a-8ba2-aa982f827bd2 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:49.564484 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:49.623206 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0f0067d-b8a8-4a8a-8ba2-aa982f827bd2 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:49.659417 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a94d436-2b45-4c0b-9493-38a20b85f804 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:49.660028 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a94d436-2b45-4c0b-9493-38a20b85f804 None admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:49.660308 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a94d436-2b45-4c0b-9493-38a20b85f804 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:49.660579 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a94d436-2b45-4c0b-9493-38a20b85f804 None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:49.661266 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a94d436-2b45-4c0b-9493-38a20b85f804 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:49.662628 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a94d436-2b45-4c0b-9493-38a20b85f804 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:49.684663 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 493/989] 10.222.0.25 () {60 vars in 1141 bytes} [Sun Sep 25 04:23:49 2022] POST /identity/v3/services => generated 242 bytes in 163 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:23:50.100251 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64a38a5e-7a57-4faa-b6a0-1aecec1b60ae None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:50.100837 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64a38a5e-7a57-4faa-b6a0-1aecec1b60ae None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:50.100837 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64a38a5e-7a57-4faa-b6a0-1aecec1b60ae None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:50.102403 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 497/990] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:23:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:23:50.108435 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bf6a42d-55be-4699-91b5-a1f772b4c31a None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:50.108755 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bf6a42d-55be-4699-91b5-a1f772b4c31a None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:50.109039 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bf6a42d-55be-4699-91b5-a1f772b4c31a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:50.129116 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-3bf6a42d-55be-4699-91b5-a1f772b4c31a None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:23:50.130293 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-3bf6a42d-55be-4699-91b5-a1f772b4c31a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:23:50.212032 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 494/991] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:23:50 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 04:23:50.486036 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64a38a5e-7a57-4faa-b6a0-1aecec1b60ae None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:50.588556 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64a38a5e-7a57-4faa-b6a0-1aecec1b60ae None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:50.624839 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b17bc033-43ce-4e30-a35a-3f4d9d8ce1fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:50.625486 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b17bc033-43ce-4e30-a35a-3f4d9d8ce1fe None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:50.625779 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b17bc033-43ce-4e30-a35a-3f4d9d8ce1fe None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:50.626017 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b17bc033-43ce-4e30-a35a-3f4d9d8ce1fe None admin] PATH_INFO: `/v3/services/image` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:50.626598 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b17bc033-43ce-4e30-a35a-3f4d9d8ce1fe None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:50.627954 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b17bc033-43ce-4e30-a35a-3f4d9d8ce1fe None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:50.636783 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-b17bc033-43ce-4e30-a35a-3f4d9d8ce1fe None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Sep 25 04:23:50.636783 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:50.636783 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 25 04:23:50.636783 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 25 04:23:50.636783 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 25 04:23:50.636783 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 25 04:23:50.636783 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 25 04:23:50.636783 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 04:23:50.636783 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 25 04:23:50.636783 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:23:50.636783 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 25 04:23:50.636783 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:23:50.636783 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:50.636783 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:50.636783 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:50.636783 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:50.636783 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:50.637892 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:50.637892 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:50.637892 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:50.637892 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:50.637892 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:50.637892 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:50.637892 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 25 04:23:50.637892 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 25 04:23:50.637892 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 25 04:23:50.637892 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 25 04:23:50.637892 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:50.637892 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:50.637892 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:50.637892 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:50.637892 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:50.637892 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:50.638780 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:50.638780 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:50.638780 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:50.638780 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:50.638780 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:50.638780 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:50.638780 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:50.638780 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:50.638780 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:50.638780 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:50.638780 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 25 04:23:50.638780 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 04:23:50.638780 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 25 04:23:50.638780 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:23:50.640991 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 498/992] 10.222.0.25 () {56 vars in 1105 bytes} [Sun Sep 25 04:23:50 2022] GET /identity/v3/services/image => generated 86 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:50.646443 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bf6a42d-55be-4699-91b5-a1f772b4c31a None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:50.672833 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:50.752098 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bf6a42d-55be-4699-91b5-a1f772b4c31a None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:50.787607 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7ed46ab-ee7a-4f14-8066-715b8ce789be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:50.788539 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7ed46ab-ee7a-4f14-8066-715b8ce789be None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:50.788989 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7ed46ab-ee7a-4f14-8066-715b8ce789be None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:50.789461 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7ed46ab-ee7a-4f14-8066-715b8ce789be None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:50.790255 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7ed46ab-ee7a-4f14-8066-715b8ce789be None admin] RBAC: Adding query filter params (name=image) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:50.790719 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7ed46ab-ee7a-4f14-8066-715b8ce789be None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:50.792246 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7ed46ab-ee7a-4f14-8066-715b8ce789be None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:50.798399 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 495/993] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:23:50 2022] GET /identity/v3/services?name=image => generated 122 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:50.803881 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b17bc033-43ce-4e30-a35a-3f4d9d8ce1fe None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:50.833696 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:50.907505 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b17bc033-43ce-4e30-a35a-3f4d9d8ce1fe None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:50.941263 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-740eb96d-612e-4b1f-b6cb-aadff8b74eea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:50.941873 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-740eb96d-612e-4b1f-b6cb-aadff8b74eea None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:50.942150 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-740eb96d-612e-4b1f-b6cb-aadff8b74eea None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:50.942420 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-740eb96d-612e-4b1f-b6cb-aadff8b74eea None admin] PATH_INFO: `/v3/services` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:50.943041 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-740eb96d-612e-4b1f-b6cb-aadff8b74eea None admin] RBAC: Adding query filter params (type=image) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:50.943342 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-740eb96d-612e-4b1f-b6cb-aadff8b74eea None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:50.944738 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-740eb96d-612e-4b1f-b6cb-aadff8b74eea None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:50.949431 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 499/994] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:23:50 2022] GET /identity/v3/services?type=image => generated 122 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:51.210724 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eccb717b-9147-4ede-99ee-19ba25a7eaa5 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:51.211089 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eccb717b-9147-4ede-99ee-19ba25a7eaa5 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:51.211291 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eccb717b-9147-4ede-99ee-19ba25a7eaa5 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:51.212660 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 496/995] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:23:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:23:51.218668 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89663df3-b56f-412c-b9d2-7cf56b4bc420 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:51.219131 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89663df3-b56f-412c-b9d2-7cf56b4bc420 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:51.219232 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89663df3-b56f-412c-b9d2-7cf56b4bc420 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:51.237908 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-89663df3-b56f-412c-b9d2-7cf56b4bc420 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:23:51.239084 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-89663df3-b56f-412c-b9d2-7cf56b4bc420 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:23:51.308676 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 500/996] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:23:51 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 04:23:51.583148 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eccb717b-9147-4ede-99ee-19ba25a7eaa5 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:51.677848 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:51.690867 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eccb717b-9147-4ede-99ee-19ba25a7eaa5 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:51.725674 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1613873-dee2-491a-9992-54345e036e1c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:51.726314 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1613873-dee2-491a-9992-54345e036e1c None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:51.726597 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1613873-dee2-491a-9992-54345e036e1c None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:51.726883 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1613873-dee2-491a-9992-54345e036e1c None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:51.727494 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1613873-dee2-491a-9992-54345e036e1c None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:51.728915 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1613873-dee2-491a-9992-54345e036e1c None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:51.738475 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-b1613873-dee2-491a-9992-54345e036e1c None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 25 04:23:51.738475 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:51.738475 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 25 04:23:51.738475 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 25 04:23:51.738475 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 25 04:23:51.738475 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 25 04:23:51.738475 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 25 04:23:51.738475 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 04:23:51.738475 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 25 04:23:51.738475 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:51.738475 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 25 04:23:51.738475 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:51.738475 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:51.738475 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:51.738475 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:51.738475 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:51.738475 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:51.739488 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:51.739488 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:51.739488 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:51.739488 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:51.739488 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:51.739488 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:51.739488 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 25 04:23:51.739488 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 25 04:23:51.739488 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 25 04:23:51.739488 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 25 04:23:51.739488 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:51.739488 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:51.739488 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:51.739488 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:51.739488 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:51.739488 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:51.740356 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:51.740356 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:51.740356 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:51.740356 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:51.740356 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:51.740356 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:51.740356 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:51.740356 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:51.740356 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:51.740356 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:51.740356 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 25 04:23:51.740356 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 04:23:51.740356 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 25 04:23:51.740356 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:51.741954 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 497/997] 10.222.0.25 () {56 vars in 1126 bytes} [Sun Sep 25 04:23:51 2022] GET /identity/v3/services/object-store => generated 93 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:51.745485 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89663df3-b56f-412c-b9d2-7cf56b4bc420 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:51.769143 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:51.853841 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89663df3-b56f-412c-b9d2-7cf56b4bc420 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:51.888399 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec382963-c85d-4d96-9433-ae47039ec0f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:51.889578 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec382963-c85d-4d96-9433-ae47039ec0f3 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:51.889854 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec382963-c85d-4d96-9433-ae47039ec0f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:51.890116 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec382963-c85d-4d96-9433-ae47039ec0f3 None admin] PATH_INFO: `/v3/services` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:51.890916 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec382963-c85d-4d96-9433-ae47039ec0f3 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:51.891215 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec382963-c85d-4d96-9433-ae47039ec0f3 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:51.892690 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec382963-c85d-4d96-9433-ae47039ec0f3 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:51.893639 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eff711d7-bd6c-4a26-ae39-fc9b70027284 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:51.893927 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eff711d7-bd6c-4a26-ae39-fc9b70027284 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:51.894200 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eff711d7-bd6c-4a26-ae39-fc9b70027284 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:51.895658 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 498/998] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:23:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:23:51.902516 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 501/999] 10.222.0.25 () {56 vars in 1122 bytes} [Sun Sep 25 04:23:51 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:51.902650 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b4ea22f-e500-4963-8c62-445028b779c5 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:51.902650 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b4ea22f-e500-4963-8c62-445028b779c5 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:51.902650 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b4ea22f-e500-4963-8c62-445028b779c5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:51.903289 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec382963-c85d-4d96-9433-ae47039ec0f3 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:51.919634 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-8b4ea22f-e500-4963-8c62-445028b779c5 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:23:51.921137 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-8b4ea22f-e500-4963-8c62-445028b779c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:23:51.995329 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:51.996033 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 499/1000] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:23:51 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 04:23:52.012469 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec382963-c85d-4d96-9433-ae47039ec0f3 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:52.047263 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b1237ce-78be-41e3-b575-126889c929f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:52.047861 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b1237ce-78be-41e3-b575-126889c929f0 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:52.048137 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b1237ce-78be-41e3-b575-126889c929f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:52.048404 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b1237ce-78be-41e3-b575-126889c929f0 None admin] PATH_INFO: `/v3/services` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:52.049050 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b1237ce-78be-41e3-b575-126889c929f0 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:52.049352 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b1237ce-78be-41e3-b575-126889c929f0 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:52.050695 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b1237ce-78be-41e3-b575-126889c929f0 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:52.060181 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 502/1001] 10.222.0.25 () {56 vars in 1122 bytes} [Sun Sep 25 04:23:51 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:23:52.061775 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b4ea22f-e500-4963-8c62-445028b779c5 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:52.149385 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:52.166250 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b4ea22f-e500-4963-8c62-445028b779c5 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:52.202793 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03f8b17e-f913-4c9e-a553-5a5a9c4d2b7a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:52.203425 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03f8b17e-f913-4c9e-a553-5a5a9c4d2b7a None admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:52.203709 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03f8b17e-f913-4c9e-a553-5a5a9c4d2b7a None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:52.203976 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03f8b17e-f913-4c9e-a553-5a5a9c4d2b7a None admin] PATH_INFO: `/v3/endpoints` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:52.204658 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03f8b17e-f913-4c9e-a553-5a5a9c4d2b7a None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:52.206024 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03f8b17e-f913-4c9e-a553-5a5a9c4d2b7a None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:52.223695 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 500/1002] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:23:52 2022] POST /identity/v3/endpoints => generated 325 bytes in 163 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:23:52.277048 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b1237ce-78be-41e3-b575-126889c929f0 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:52.390092 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b1237ce-78be-41e3-b575-126889c929f0 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:52.433436 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e41a3c8-a126-4046-8b73-27b7bfca5f79 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:52.434073 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e41a3c8-a126-4046-8b73-27b7bfca5f79 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:52.434353 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e41a3c8-a126-4046-8b73-27b7bfca5f79 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:52.434617 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e41a3c8-a126-4046-8b73-27b7bfca5f79 None admin] PATH_INFO: `/v3/services/compute` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:52.435201 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e41a3c8-a126-4046-8b73-27b7bfca5f79 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:52.437145 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e41a3c8-a126-4046-8b73-27b7bfca5f79 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:52.445900 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-7e41a3c8-a126-4046-8b73-27b7bfca5f79 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Sep 25 04:23:52.445900 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:52.445900 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 25 04:23:52.445900 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 25 04:23:52.445900 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 25 04:23:52.445900 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 25 04:23:52.445900 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 25 04:23:52.445900 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 04:23:52.445900 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 25 04:23:52.445900 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:23:52.445900 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 25 04:23:52.445900 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:23:52.445900 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:52.445900 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:52.445900 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:52.445900 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:52.445900 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:52.446917 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:52.446917 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:52.446917 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:52.446917 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:52.446917 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:52.446917 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:52.446917 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 25 04:23:52.446917 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 25 04:23:52.446917 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 25 04:23:52.446917 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 25 04:23:52.446917 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:52.446917 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:52.446917 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:52.446917 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:52.446917 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:52.446917 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:52.447791 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:52.447791 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:52.447791 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:52.447791 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:52.447791 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:52.447791 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:52.447791 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:52.447791 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:52.447791 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:52.447791 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:52.447791 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 25 04:23:52.447791 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 04:23:52.447791 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 25 04:23:52.447791 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:23:52.447791 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 503/1003] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Sep 25 04:23:52 2022] GET /identity/v3/services/compute => generated 88 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:52.455138 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03f8b17e-f913-4c9e-a553-5a5a9c4d2b7a None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:52.561817 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03f8b17e-f913-4c9e-a553-5a5a9c4d2b7a None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:52.596258 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-964dd132-69d1-43cd-9b72-67eb38187d23 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:52.596909 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-964dd132-69d1-43cd-9b72-67eb38187d23 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:52.597194 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-964dd132-69d1-43cd-9b72-67eb38187d23 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:52.597464 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-964dd132-69d1-43cd-9b72-67eb38187d23 None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:52.598089 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-964dd132-69d1-43cd-9b72-67eb38187d23 None admin] RBAC: Adding query filter params (name=compute) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:52.598394 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-964dd132-69d1-43cd-9b72-67eb38187d23 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:52.599769 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-964dd132-69d1-43cd-9b72-67eb38187d23 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:52.604860 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 501/1004] 10.222.0.25 () {56 vars in 1112 bytes} [Sun Sep 25 04:23:52 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:52.610893 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e41a3c8-a126-4046-8b73-27b7bfca5f79 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:52.622487 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:52.715219 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e41a3c8-a126-4046-8b73-27b7bfca5f79 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:52.750633 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5db473fd-210c-442c-ae88-b9389c373afa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:52.751251 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5db473fd-210c-442c-ae88-b9389c373afa None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:52.751533 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5db473fd-210c-442c-ae88-b9389c373afa None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:52.751807 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5db473fd-210c-442c-ae88-b9389c373afa None admin] PATH_INFO: `/v3/services` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:52.752435 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5db473fd-210c-442c-ae88-b9389c373afa None admin] RBAC: Adding query filter params (type=compute) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:52.752770 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5db473fd-210c-442c-ae88-b9389c373afa None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:52.754131 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5db473fd-210c-442c-ae88-b9389c373afa None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:52.759192 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 504/1005] 10.222.0.25 () {56 vars in 1112 bytes} [Sun Sep 25 04:23:52 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:52.765578 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-964dd132-69d1-43cd-9b72-67eb38187d23 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:52.777356 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:52.869252 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-964dd132-69d1-43cd-9b72-67eb38187d23 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:52.902850 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-047149ec-7e67-4be7-86b6-8b458727a6af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:52.903467 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-047149ec-7e67-4be7-86b6-8b458727a6af None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:52.903752 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-047149ec-7e67-4be7-86b6-8b458727a6af None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:52.904024 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-047149ec-7e67-4be7-86b6-8b458727a6af None admin] PATH_INFO: `/v3/endpoints` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:52.905419 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-047149ec-7e67-4be7-86b6-8b458727a6af None admin] RBAC: Adding query filter params (interface=public, service_id=028f3e6cf6174e37a92539ae84d1d231, region_id=RegionOne) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:52.905741 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-047149ec-7e67-4be7-86b6-8b458727a6af None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:52.907107 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-047149ec-7e67-4be7-86b6-8b458727a6af None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:52.917063 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 502/1006] 10.222.0.25 () {56 vars in 1251 bytes} [Sun Sep 25 04:23:52 2022] GET /identity/v3/endpoints?service_id=028f3e6cf6174e37a92539ae84d1d231&interface=public®ion_id=RegionOne => generated 194 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:52.918029 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5db473fd-210c-442c-ae88-b9389c373afa None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:52.990410 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:53.021984 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5db473fd-210c-442c-ae88-b9389c373afa None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:53.059955 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d61968a3-e721-421f-9f8b-caf9efb60829 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:53.060582 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d61968a3-e721-421f-9f8b-caf9efb60829 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:53.060925 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d61968a3-e721-421f-9f8b-caf9efb60829 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:53.061200 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d61968a3-e721-421f-9f8b-caf9efb60829 None admin] PATH_INFO: `/v3/services` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:53.061833 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d61968a3-e721-421f-9f8b-caf9efb60829 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:53.063204 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d61968a3-e721-421f-9f8b-caf9efb60829 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:53.077045 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 505/1007] 10.222.0.25 () {56 vars in 1087 bytes} [Sun Sep 25 04:23:52 2022] GET /identity/v3/services => generated 1727 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 25 04:23:53.160384 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2338ce6-eabc-4528-886b-80abc102f02b None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:53.160748 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2338ce6-eabc-4528-886b-80abc102f02b None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:53.160984 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2338ce6-eabc-4528-886b-80abc102f02b None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:53.162541 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 503/1008] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:23:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 25 04:23:53.168268 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9e83643-ee51-4207-8a70-265757076793 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:53.168560 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9e83643-ee51-4207-8a70-265757076793 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:53.168855 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9e83643-ee51-4207-8a70-265757076793 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:53.189237 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-f9e83643-ee51-4207-8a70-265757076793 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:23:53.190474 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-f9e83643-ee51-4207-8a70-265757076793 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:23:53.265403 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 506/1009] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:23:53 2022] POST /identity/v3/auth/tokens => generated 1717 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 04:23:53.530283 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2338ce6-eabc-4528-886b-80abc102f02b None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:53.608599 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:53.642102 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2338ce6-eabc-4528-886b-80abc102f02b None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:53.676050 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dedce0b6-9c6d-4764-951f-b7ea4e887643 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:53.676682 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dedce0b6-9c6d-4764-951f-b7ea4e887643 None admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:53.676972 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dedce0b6-9c6d-4764-951f-b7ea4e887643 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:53.677250 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dedce0b6-9c6d-4764-951f-b7ea4e887643 None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:53.677911 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dedce0b6-9c6d-4764-951f-b7ea4e887643 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:53.679271 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dedce0b6-9c6d-4764-951f-b7ea4e887643 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:53.696685 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 504/1010] 10.222.0.25 () {60 vars in 1141 bytes} [Sun Sep 25 04:23:53 2022] POST /identity/v3/services => generated 242 bytes in 165 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 04:23:54.431000 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c73c4a4-bb63-45f8-b759-9210c8e85766 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:54.431527 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c73c4a4-bb63-45f8-b759-9210c8e85766 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:54.431527 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c73c4a4-bb63-45f8-b759-9210c8e85766 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:54.433066 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 507/1011] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:23:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:23:54.438945 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7061133-412e-4b96-a242-da33c3a14bf7 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:54.439380 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7061133-412e-4b96-a242-da33c3a14bf7 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:54.439525 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7061133-412e-4b96-a242-da33c3a14bf7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:54.458819 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-e7061133-412e-4b96-a242-da33c3a14bf7 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:23:54.459995 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-e7061133-412e-4b96-a242-da33c3a14bf7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:23:54.537234 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 505/1012] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:23:54 2022] POST /identity/v3/auth/tokens => generated 1813 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 04:23:54.804440 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c73c4a4-bb63-45f8-b759-9210c8e85766 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:54.869950 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:54.905574 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c73c4a4-bb63-45f8-b759-9210c8e85766 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:54.940671 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88a015a2-2c7f-44b2-9a66-91d963afd705 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:54.941308 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88a015a2-2c7f-44b2-9a66-91d963afd705 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:54.941586 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88a015a2-2c7f-44b2-9a66-91d963afd705 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:54.941859 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88a015a2-2c7f-44b2-9a66-91d963afd705 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:54.942983 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88a015a2-2c7f-44b2-9a66-91d963afd705 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:54.944403 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88a015a2-2c7f-44b2-9a66-91d963afd705 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:54.946185 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 508/1013] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:23:54 2022] GET /identity/v3/domains/default => generated 199 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:54.952386 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7061133-412e-4b96-a242-da33c3a14bf7 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:55.013490 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:55.056894 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7061133-412e-4b96-a242-da33c3a14bf7 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:55.090297 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5e9a348-fccf-4fe0-bd10-d3291317521b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:55.090957 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5e9a348-fccf-4fe0-bd10-d3291317521b None admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:55.091207 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5e9a348-fccf-4fe0-bd10-d3291317521b None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:55.091519 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5e9a348-fccf-4fe0-bd10-d3291317521b None admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:55.092789 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5e9a348-fccf-4fe0-bd10-d3291317521b None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:55.094214 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5e9a348-fccf-4fe0-bd10-d3291317521b None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:55.120721 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 506/1014] 10.222.0.25 () {60 vars in 1141 bytes} [Sun Sep 25 04:23:54 2022] POST /identity/v3/projects => generated 311 bytes in 169 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:23:55.288460 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc7e3940-a915-4fee-9d0f-43a0e04abffe None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:55.288827 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc7e3940-a915-4fee-9d0f-43a0e04abffe None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:55.289027 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc7e3940-a915-4fee-9d0f-43a0e04abffe None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:55.292301 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 509/1015] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:23:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:23:55.295916 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5e28ed4-94a0-4eb2-b74d-8fbf57d1eba8 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:55.296216 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5e28ed4-94a0-4eb2-b74d-8fbf57d1eba8 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:55.296484 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5e28ed4-94a0-4eb2-b74d-8fbf57d1eba8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:55.315776 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-c5e28ed4-94a0-4eb2-b74d-8fbf57d1eba8 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:23:55.317356 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-c5e28ed4-94a0-4eb2-b74d-8fbf57d1eba8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:23:55.392682 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 507/1016] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:23:55 2022] POST /identity/v3/auth/tokens => generated 1813 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 04:23:55.659338 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc7e3940-a915-4fee-9d0f-43a0e04abffe None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:55.767151 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc7e3940-a915-4fee-9d0f-43a0e04abffe None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:55.804130 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e0092af-798f-4bbd-84fd-b8236d385c72 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:55.804812 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e0092af-798f-4bbd-84fd-b8236d385c72 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:55.805098 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e0092af-798f-4bbd-84fd-b8236d385c72 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:55.805368 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e0092af-798f-4bbd-84fd-b8236d385c72 None admin] PATH_INFO: `/v3/services/compute` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:55.805950 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e0092af-798f-4bbd-84fd-b8236d385c72 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:55.807819 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e0092af-798f-4bbd-84fd-b8236d385c72 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:55.809669 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:55.818966 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-6e0092af-798f-4bbd-84fd-b8236d385c72 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Sep 25 04:23:55.818966 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:55.818966 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 25 04:23:55.818966 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 25 04:23:55.818966 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 25 04:23:55.818966 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 25 04:23:55.818966 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 25 04:23:55.818966 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 04:23:55.818966 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 25 04:23:55.818966 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:23:55.818966 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 25 04:23:55.818966 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:23:55.818966 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:55.818966 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:55.818966 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:55.818966 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:55.818966 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:55.820021 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:55.820021 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:55.820021 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:55.820021 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:55.820021 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:55.820021 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:55.820021 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 25 04:23:55.820021 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 25 04:23:55.820021 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 25 04:23:55.820021 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 25 04:23:55.820021 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:55.820021 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:55.820021 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:55.820021 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:55.820021 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:55.820021 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:55.821344 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:55.821344 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:55.821344 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:55.821344 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:55.821344 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:55.821344 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:55.821344 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:55.821344 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:55.821344 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:55.821344 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:55.821344 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 25 04:23:55.821344 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 04:23:55.821344 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 25 04:23:55.821344 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:23:55.824996 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 510/1017] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Sep 25 04:23:55 2022] GET /identity/v3/services/compute => generated 88 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:55.826266 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5e28ed4-94a0-4eb2-b74d-8fbf57d1eba8 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:55.868263 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:55.880546 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad45d7b8-ab80-4bcf-b604-548e0fec5a9f None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:55.880892 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad45d7b8-ab80-4bcf-b604-548e0fec5a9f None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:55.881156 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad45d7b8-ab80-4bcf-b604-548e0fec5a9f None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:55.882621 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 511/1018] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:23:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 25 04:23:55.887528 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b4e2133-3a4e-4dd8-a59b-18de43052645 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:55.887817 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b4e2133-3a4e-4dd8-a59b-18de43052645 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:55.888091 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b4e2133-3a4e-4dd8-a59b-18de43052645 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:55.907730 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-3b4e2133-3a4e-4dd8-a59b-18de43052645 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:23:55.908990 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-3b4e2133-3a4e-4dd8-a59b-18de43052645 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:23:55.932857 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5e28ed4-94a0-4eb2-b74d-8fbf57d1eba8 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:55.969402 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6b01200-4c27-447f-8d28-23dd89ef30c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:55.970025 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b01200-4c27-447f-8d28-23dd89ef30c1 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:55.970305 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b01200-4c27-447f-8d28-23dd89ef30c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:55.970578 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b01200-4c27-447f-8d28-23dd89ef30c1 None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:55.971204 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6b01200-4c27-447f-8d28-23dd89ef30c1 None admin] RBAC: Adding query filter params (name=compute) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:55.971508 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6b01200-4c27-447f-8d28-23dd89ef30c1 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:55.972933 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6b01200-4c27-447f-8d28-23dd89ef30c1 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:55.977785 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 508/1019] 10.222.0.25 () {56 vars in 1112 bytes} [Sun Sep 25 04:23:55 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:23:55.983308 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6b01200-4c27-447f-8d28-23dd89ef30c1 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:55.984056 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 512/1020] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:23:55 2022] POST /identity/v3/auth/tokens => generated 1813 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 25 04:23:56.085228 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6b01200-4c27-447f-8d28-23dd89ef30c1 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:56.087572 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:56.121609 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1cc2d14-a2c3-46a7-86ad-b9e7f36abbeb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:56.122222 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1cc2d14-a2c3-46a7-86ad-b9e7f36abbeb None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:56.122507 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1cc2d14-a2c3-46a7-86ad-b9e7f36abbeb None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:56.122784 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1cc2d14-a2c3-46a7-86ad-b9e7f36abbeb None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:56.123462 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1cc2d14-a2c3-46a7-86ad-b9e7f36abbeb None admin] RBAC: Adding query filter params (type=compute) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:56.123773 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1cc2d14-a2c3-46a7-86ad-b9e7f36abbeb None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:56.125178 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1cc2d14-a2c3-46a7-86ad-b9e7f36abbeb None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:56.134744 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 509/1021] 10.222.0.25 () {56 vars in 1112 bytes} [Sun Sep 25 04:23:55 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:23:56.135558 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b4e2133-3a4e-4dd8-a59b-18de43052645 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:56.233633 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b4e2133-3a4e-4dd8-a59b-18de43052645 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:56.252409 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1cc2d14-a2c3-46a7-86ad-b9e7f36abbeb None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:56.259821 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:56.269856 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68705d3c-4bf8-4732-8b40-b6f9f55b7ecf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:56.270469 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68705d3c-4bf8-4732-8b40-b6f9f55b7ecf None admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:56.270751 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68705d3c-4bf8-4732-8b40-b6f9f55b7ecf None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:56.271023 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68705d3c-4bf8-4732-8b40-b6f9f55b7ecf None admin] PATH_INFO: `/v3/endpoints` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:56.271706 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68705d3c-4bf8-4732-8b40-b6f9f55b7ecf None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:56.274434 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68705d3c-4bf8-4732-8b40-b6f9f55b7ecf None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:56.297480 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 513/1022] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:23:56 2022] POST /identity/v3/endpoints => generated 334 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 04:23:56.370056 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1cc2d14-a2c3-46a7-86ad-b9e7f36abbeb None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:56.416421 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea0f391d-ffc4-4c71-ae83-46ead9238212 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:56.417387 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea0f391d-ffc4-4c71-ae83-46ead9238212 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:56.417821 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea0f391d-ffc4-4c71-ae83-46ead9238212 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:56.418243 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea0f391d-ffc4-4c71-ae83-46ead9238212 None admin] PATH_INFO: `/v3/services/image` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:56.418987 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea0f391d-ffc4-4c71-ae83-46ead9238212 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:56.420666 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea0f391d-ffc4-4c71-ae83-46ead9238212 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:56.430819 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-ea0f391d-ffc4-4c71-ae83-46ead9238212 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Sep 25 04:23:56.430819 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:56.430819 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 25 04:23:56.430819 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 25 04:23:56.430819 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 25 04:23:56.430819 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 25 04:23:56.430819 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 25 04:23:56.430819 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 04:23:56.430819 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 25 04:23:56.430819 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:56.430819 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 25 04:23:56.430819 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:56.430819 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:56.430819 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:56.430819 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:56.430819 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:56.430819 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:56.432177 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:56.432177 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:56.432177 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:56.432177 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:56.432177 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:56.432177 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:56.432177 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 25 04:23:56.432177 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 25 04:23:56.432177 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 25 04:23:56.432177 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 25 04:23:56.432177 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:56.432177 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:56.432177 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:56.432177 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:56.432177 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:56.432177 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:56.433296 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:56.433296 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:56.433296 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:56.433296 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:56.433296 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:56.433296 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:56.433296 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:56.433296 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:56.433296 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:56.433296 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:56.433296 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 25 04:23:56.433296 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 04:23:56.433296 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 25 04:23:56.433296 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:56.435534 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 510/1023] 10.222.0.25 () {56 vars in 1105 bytes} [Sun Sep 25 04:23:56 2022] GET /identity/v3/services/image => generated 86 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 04:23:56.442218 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68705d3c-4bf8-4732-8b40-b6f9f55b7ecf None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:56.561725 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68705d3c-4bf8-4732-8b40-b6f9f55b7ecf None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:56.599537 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbceb154-5c67-419d-8c41-290036434644 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:56.600152 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbceb154-5c67-419d-8c41-290036434644 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:56.600434 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbceb154-5c67-419d-8c41-290036434644 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:56.600733 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbceb154-5c67-419d-8c41-290036434644 None admin] PATH_INFO: `/v3/services` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:56.601361 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbceb154-5c67-419d-8c41-290036434644 None admin] RBAC: Adding query filter params (name=image) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:56.601662 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbceb154-5c67-419d-8c41-290036434644 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:56.603121 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbceb154-5c67-419d-8c41-290036434644 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:56.608740 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 514/1024] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:23:56 2022] GET /identity/v3/services?name=image => generated 122 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:56.614243 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea0f391d-ffc4-4c71-ae83-46ead9238212 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:56.643692 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:56.718590 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea0f391d-ffc4-4c71-ae83-46ead9238212 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:56.753663 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e6c6fac-a81c-4c6c-98be-834138e69479 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:56.754426 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e6c6fac-a81c-4c6c-98be-834138e69479 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:56.754811 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e6c6fac-a81c-4c6c-98be-834138e69479 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:56.755185 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e6c6fac-a81c-4c6c-98be-834138e69479 None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:56.755912 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e6c6fac-a81c-4c6c-98be-834138e69479 None admin] RBAC: Adding query filter params (type=image) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:56.756316 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e6c6fac-a81c-4c6c-98be-834138e69479 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:56.757869 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e6c6fac-a81c-4c6c-98be-834138e69479 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:56.767599 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 511/1025] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:23:56 2022] GET /identity/v3/services?type=image => generated 350 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:56.768890 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbceb154-5c67-419d-8c41-290036434644 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:56.819870 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:56.873059 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbceb154-5c67-419d-8c41-290036434644 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:56.907766 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faf13bcc-79ed-4d16-b3e7-e5803f2dddbb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:56.908378 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf13bcc-79ed-4d16-b3e7-e5803f2dddbb None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:56.908674 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf13bcc-79ed-4d16-b3e7-e5803f2dddbb None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:56.908951 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf13bcc-79ed-4d16-b3e7-e5803f2dddbb None admin] PATH_INFO: `/v3/endpoints` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:56.909649 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faf13bcc-79ed-4d16-b3e7-e5803f2dddbb None admin] RBAC: Adding query filter params (interface=public, service_id=def80176fc6e498faf487d82b2c5d52d, region_id=RegionOne) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:56.909954 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faf13bcc-79ed-4d16-b3e7-e5803f2dddbb None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:56.911304 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faf13bcc-79ed-4d16-b3e7-e5803f2dddbb None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:56.921979 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 515/1026] 10.222.0.25 () {56 vars in 1251 bytes} [Sun Sep 25 04:23:56 2022] GET /identity/v3/endpoints?service_id=def80176fc6e498faf487d82b2c5d52d&interface=public®ion_id=RegionOne => generated 194 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:56.923220 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e6c6fac-a81c-4c6c-98be-834138e69479 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:57.014770 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:57.024767 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e6c6fac-a81c-4c6c-98be-834138e69479 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:57.058144 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e6eea8c-278a-42a5-864f-541eac45b42d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:57.058750 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e6eea8c-278a-42a5-864f-541eac45b42d None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:57.059026 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e6eea8c-278a-42a5-864f-541eac45b42d None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:57.059290 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e6eea8c-278a-42a5-864f-541eac45b42d None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:57.059905 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e6eea8c-278a-42a5-864f-541eac45b42d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:57.061353 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e6eea8c-278a-42a5-864f-541eac45b42d None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:57.077054 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 512/1027] 10.222.0.25 () {56 vars in 1087 bytes} [Sun Sep 25 04:23:56 2022] GET /identity/v3/services => generated 1957 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 25 04:23:57.370862 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b99859fe-617b-47a6-be5f-af55b4b4a3ce None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:57.371045 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b99859fe-617b-47a6-be5f-af55b4b4a3ce None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:57.371325 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b99859fe-617b-47a6-be5f-af55b4b4a3ce None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:57.372695 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 516/1028] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:23:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:23:57.378579 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c58befc-63ed-4751-8a7f-cc6d4de47b04 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:57.378956 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c58befc-63ed-4751-8a7f-cc6d4de47b04 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:57.379318 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c58befc-63ed-4751-8a7f-cc6d4de47b04 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:57.397518 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-5c58befc-63ed-4751-8a7f-cc6d4de47b04 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:23:57.398697 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-5c58befc-63ed-4751-8a7f-cc6d4de47b04 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:23:57.466204 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 513/1029] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:23:57 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 04:23:57.736949 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b99859fe-617b-47a6-be5f-af55b4b4a3ce None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:57.838497 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b99859fe-617b-47a6-be5f-af55b4b4a3ce None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:57.851597 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:57.873392 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f2f34b7-1737-405d-ae52-ebcebaf3cae1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:57.874013 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f2f34b7-1737-405d-ae52-ebcebaf3cae1 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:57.874289 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f2f34b7-1737-405d-ae52-ebcebaf3cae1 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:57.874557 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f2f34b7-1737-405d-ae52-ebcebaf3cae1 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:57.875663 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f2f34b7-1737-405d-ae52-ebcebaf3cae1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:57.877145 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f2f34b7-1737-405d-ae52-ebcebaf3cae1 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:57.878889 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 517/1030] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:23:57 2022] GET /identity/v3/domains/default => generated 199 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:57.884521 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c58befc-63ed-4751-8a7f-cc6d4de47b04 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:57.971999 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:57.989587 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c58befc-63ed-4751-8a7f-cc6d4de47b04 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:58.024919 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6ab8c64-ee24-41cf-a4f2-d0b5227b1e33 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:58.025529 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6ab8c64-ee24-41cf-a4f2-d0b5227b1e33 None admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:58.025807 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6ab8c64-ee24-41cf-a4f2-d0b5227b1e33 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:58.026078 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6ab8c64-ee24-41cf-a4f2-d0b5227b1e33 None admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:58.026745 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6ab8c64-ee24-41cf-a4f2-d0b5227b1e33 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:58.028167 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6ab8c64-ee24-41cf-a4f2-d0b5227b1e33 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:58.060530 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 514/1031] 10.222.0.25 () {60 vars in 1132 bytes} [Sun Sep 25 04:23:57 2022] POST /identity/v3/users => generated 285 bytes in 169 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:23:58.611903 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd85692-0209-415e-97ff-1c4cf1eef123 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:58.612220 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd85692-0209-415e-97ff-1c4cf1eef123 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:58.612458 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd85692-0209-415e-97ff-1c4cf1eef123 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:58.613879 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 518/1032] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:23:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:23:58.618989 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae387f54-42e2-4d4a-97d0-4becc2cd2ea9 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:58.619287 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae387f54-42e2-4d4a-97d0-4becc2cd2ea9 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:58.619566 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae387f54-42e2-4d4a-97d0-4becc2cd2ea9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:58.637981 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-ae387f54-42e2-4d4a-97d0-4becc2cd2ea9 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:23:58.639721 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-ae387f54-42e2-4d4a-97d0-4becc2cd2ea9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:23:58.715839 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 515/1033] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:23:58 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 04:23:58.989193 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcd85692-0209-415e-97ff-1c4cf1eef123 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:59.095004 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcd85692-0209-415e-97ff-1c4cf1eef123 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:59.130174 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f204eaea-b885-4144-a306-3424d538ee61 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:59.131001 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f204eaea-b885-4144-a306-3424d538ee61 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:59.131283 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f204eaea-b885-4144-a306-3424d538ee61 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:59.131553 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f204eaea-b885-4144-a306-3424d538ee61 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:59.145850 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f204eaea-b885-4144-a306-3424d538ee61 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 04:23:59.150946 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-f204eaea-b885-4144-a306-3424d538ee61 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 25 04:23:59.150946 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:59.150946 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:59.150946 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:59.150946 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:59.150946 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:59.150946 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:59.150946 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:59.150946 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:59.150946 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:59.150946 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:59.150946 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:59.150946 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 04:23:59.150946 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 04:23:59.150946 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 04:23:59.150946 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 04:23:59.151981 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 04:23:59.151981 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 04:23:59.151981 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 04:23:59.151981 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 04:23:59.151981 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:59.151981 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:59.151981 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:59.151981 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:59.151981 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:59.151981 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:59.151981 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:59.151981 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:59.151981 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:59.151981 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:59.151981 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:59.151981 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:59.153429 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:59.153429 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:59.153429 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:59.153429 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:59.153429 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 04:23:59.153429 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 04:23:59.153429 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 04:23:59.153429 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 04:23:59.153429 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 04:23:59.153429 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 04:23:59.153429 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 25 04:23:59.153429 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:23:59.154543 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 519/1034] 10.222.0.25 () {56 vars in 1120 bytes} [Sun Sep 25 04:23:58 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:59.158693 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae387f54-42e2-4d4a-97d0-4becc2cd2ea9 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:59.233795 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:59.265373 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01836fb1-39ee-43c1-9216-d85be1792fc9 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:59.265687 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01836fb1-39ee-43c1-9216-d85be1792fc9 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:59.265853 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01836fb1-39ee-43c1-9216-d85be1792fc9 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:59.266081 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae387f54-42e2-4d4a-97d0-4becc2cd2ea9 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:59.267359 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 520/1035] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:23:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:23:59.272155 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6ed56f7-5976-4c55-8131-de2464e1f3a3 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:59.272436 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6ed56f7-5976-4c55-8131-de2464e1f3a3 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:59.272734 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6ed56f7-5976-4c55-8131-de2464e1f3a3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:59.291022 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-e6ed56f7-5976-4c55-8131-de2464e1f3a3 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:23:59.292208 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-e6ed56f7-5976-4c55-8131-de2464e1f3a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:23:59.304209 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a48cdca7-6a66-40e2-b138-760a26f67d38 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:59.305124 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a48cdca7-6a66-40e2-b138-760a26f67d38 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:59.305407 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a48cdca7-6a66-40e2-b138-760a26f67d38 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:59.305678 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a48cdca7-6a66-40e2-b138-760a26f67d38 None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:59.306501 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a48cdca7-6a66-40e2-b138-760a26f67d38 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:59.307189 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a48cdca7-6a66-40e2-b138-760a26f67d38 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:59.308565 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a48cdca7-6a66-40e2-b138-760a26f67d38 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:59.321838 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 516/1036] 10.222.0.25 () {56 vars in 1115 bytes} [Sun Sep 25 04:23:59 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:59.322938 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a48cdca7-6a66-40e2-b138-760a26f67d38 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:59.333770 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:59.370798 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 521/1037] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:23:59 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 25 04:23:59.429314 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a48cdca7-6a66-40e2-b138-760a26f67d38 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:59.463302 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3cdef1e-01ae-4d26-9e9b-b912e697b493 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:59.464155 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3cdef1e-01ae-4d26-9e9b-b912e697b493 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:59.464579 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3cdef1e-01ae-4d26-9e9b-b912e697b493 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:59.465010 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3cdef1e-01ae-4d26-9e9b-b912e697b493 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:59.470225 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:59.481878 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3cdef1e-01ae-4d26-9e9b-b912e697b493 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:59.484334 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3cdef1e-01ae-4d26-9e9b-b912e697b493 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:59.499084 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-b3cdef1e-01ae-4d26-9e9b-b912e697b493 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:23:59.499084 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:59.499084 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:59.499084 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:59.499084 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:59.499084 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:59.499084 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:59.499084 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:59.499084 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:59.499084 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:59.499084 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:59.499084 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:59.499084 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 04:23:59.499084 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 04:23:59.499084 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 04:23:59.499084 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 04:23:59.500062 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:59.500062 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:59.500062 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:59.500062 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:59.500062 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:59.500062 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:59.500062 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:59.500062 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:59.500062 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:59.500062 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:59.500062 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:59.500062 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:59.500062 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:59.500062 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:59.500062 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:59.500062 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:59.500062 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 04:23:59.501562 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 04:23:59.501562 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:23:59.501562 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:59.504979 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 517/1038] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:23:59 2022] GET /identity/v3/domains/Default => generated 87 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:23:59.506122 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6ed56f7-5976-4c55-8131-de2464e1f3a3 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:59.606855 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6ed56f7-5976-4c55-8131-de2464e1f3a3 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:59.642302 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07c0b6af-3c7e-48bb-ae0c-b8dce12fa57d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:59.642917 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07c0b6af-3c7e-48bb-ae0c-b8dce12fa57d None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:59.643190 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07c0b6af-3c7e-48bb-ae0c-b8dce12fa57d None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:59.643457 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07c0b6af-3c7e-48bb-ae0c-b8dce12fa57d None admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:59.644093 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07c0b6af-3c7e-48bb-ae0c-b8dce12fa57d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:59.644392 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07c0b6af-3c7e-48bb-ae0c-b8dce12fa57d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:59.645786 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07c0b6af-3c7e-48bb-ae0c-b8dce12fa57d None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:59.647727 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3cdef1e-01ae-4d26-9e9b-b912e697b493 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:59.661003 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 522/1039] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Sep 25 04:23:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:59.663543 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07c0b6af-3c7e-48bb-ae0c-b8dce12fa57d None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:59.695943 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:59.757512 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3cdef1e-01ae-4d26-9e9b-b912e697b493 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:59.773344 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07c0b6af-3c7e-48bb-ae0c-b8dce12fa57d None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:59.793047 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b801e12e-443a-4e4a-ab1a-29a6932e927a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:59.793692 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b801e12e-443a-4e4a-ab1a-29a6932e927a None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:59.793981 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b801e12e-443a-4e4a-ab1a-29a6932e927a None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:59.794251 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b801e12e-443a-4e4a-ab1a-29a6932e927a None admin] PATH_INFO: `/v3/services/image` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:59.794834 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b801e12e-443a-4e4a-ab1a-29a6932e927a None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:59.796804 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b801e12e-443a-4e4a-ab1a-29a6932e927a None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:59.805628 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-b801e12e-443a-4e4a-ab1a-29a6932e927a None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Sep 25 04:23:59.805628 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:59.805628 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 25 04:23:59.805628 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 25 04:23:59.805628 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 25 04:23:59.805628 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 25 04:23:59.805628 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 25 04:23:59.805628 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 04:23:59.805628 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 25 04:23:59.805628 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:59.805628 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 25 04:23:59.805628 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:59.805628 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:59.805628 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:59.805628 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:59.805628 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:59.805628 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:59.806664 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:59.806664 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:59.806664 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:59.806664 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:59.806664 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:59.806664 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:59.806664 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 25 04:23:59.806664 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 25 04:23:59.806664 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 25 04:23:59.806664 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 25 04:23:59.806664 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:59.806664 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:59.806664 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:59.806664 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:59.806664 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:59.806664 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:59.807534 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:59.807534 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:59.807534 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:59.807534 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:59.807534 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:59.807534 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:59.807534 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:59.807534 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:59.807534 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:59.807534 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:59.807534 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 25 04:23:59.807534 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 04:23:59.807534 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 25 04:23:59.807534 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:23:59.807534 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 518/1040] 10.222.0.25 () {56 vars in 1105 bytes} [Sun Sep 25 04:23:59 2022] GET /identity/v3/services/image => generated 86 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 04:23:59.812823 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b801e12e-443a-4e4a-ab1a-29a6932e927a None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:59.812965 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2332603e-5a56-4969-b77d-ae0a99623773 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:59.813527 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2332603e-5a56-4969-b77d-ae0a99623773 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:59.813836 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2332603e-5a56-4969-b77d-ae0a99623773 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:59.814108 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2332603e-5a56-4969-b77d-ae0a99623773 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:59.822970 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2332603e-5a56-4969-b77d-ae0a99623773 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:59.824402 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2332603e-5a56-4969-b77d-ae0a99623773 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:59.836744 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-2332603e-5a56-4969-b77d-ae0a99623773 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 25 04:23:59.836744 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:23:59.836744 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:23:59.836744 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:23:59.836744 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:23:59.836744 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:23:59.836744 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:23:59.836744 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:23:59.836744 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:23:59.836744 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:23:59.836744 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:23:59.836744 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:23:59.836744 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 25 04:23:59.836744 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 25 04:23:59.836744 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 25 04:23:59.836744 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 25 04:23:59.837739 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:23:59.837739 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:23:59.837739 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 25 04:23:59.837739 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 04:23:59.837739 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 25 04:23:59.837739 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 04:23:59.837739 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:23:59.837739 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:23:59.837739 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:23:59.837739 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:23:59.837739 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:23:59.837739 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:23:59.837739 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:23:59.837739 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:23:59.837739 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:23:59.837739 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:23:59.837739 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:23:59.837739 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:23:59.838807 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:23:59.838807 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:23:59.838807 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 25 04:23:59.838807 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 25 04:23:59.838807 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 25 04:23:59.838807 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 25 04:23:59.838807 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 25 04:23:59.838807 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 25 04:23:59.838807 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 25 04:23:59.838807 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:23:59.838807 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 523/1041] 10.222.0.25 () {56 vars in 1093 bytes} [Sun Sep 25 04:23:59 2022] GET /identity/v3/users/nova => generated 82 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 04:23:59.843658 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2332603e-5a56-4969-b77d-ae0a99623773 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:59.870777 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:59.921211 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b801e12e-443a-4e4a-ab1a-29a6932e927a None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:59.941689 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:23:59.952620 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2332603e-5a56-4969-b77d-ae0a99623773 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:23:59.958595 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e303b22a-53fe-4e23-9c34-1c1a6293a3de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:59.959209 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e303b22a-53fe-4e23-9c34-1c1a6293a3de None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:59.959486 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e303b22a-53fe-4e23-9c34-1c1a6293a3de None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:59.959764 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e303b22a-53fe-4e23-9c34-1c1a6293a3de None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:59.960382 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e303b22a-53fe-4e23-9c34-1c1a6293a3de None admin] RBAC: Adding query filter params (name=image) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:59.960712 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e303b22a-53fe-4e23-9c34-1c1a6293a3de None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:59.962077 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e303b22a-53fe-4e23-9c34-1c1a6293a3de None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:23:59.967279 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 519/1042] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:23:59 2022] GET /identity/v3/services?name=image => generated 122 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:23:59.972424 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e303b22a-53fe-4e23-9c34-1c1a6293a3de None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:23:59.990308 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8de25ea8-f69d-4324-953a-3a4a40d481be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:23:59.990969 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8de25ea8-f69d-4324-953a-3a4a40d481be None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:23:59.991187 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8de25ea8-f69d-4324-953a-3a4a40d481be None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:23:59.991455 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8de25ea8-f69d-4324-953a-3a4a40d481be None admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:23:59.992177 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8de25ea8-f69d-4324-953a-3a4a40d481be None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:23:59.992479 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8de25ea8-f69d-4324-953a-3a4a40d481be None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:23:59.993946 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8de25ea8-f69d-4324-953a-3a4a40d481be None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:00.012357 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 524/1043] 10.222.0.25 () {56 vars in 1133 bytes} [Sun Sep 25 04:23:59 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:24:00.017863 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8de25ea8-f69d-4324-953a-3a4a40d481be None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:00.082991 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e303b22a-53fe-4e23-9c34-1c1a6293a3de None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:00.097104 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:24:00.120474 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebfc4926-f9e9-47c0-8d60-3874ddacab25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:00.121180 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebfc4926-f9e9-47c0-8d60-3874ddacab25 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:00.121405 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebfc4926-f9e9-47c0-8d60-3874ddacab25 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:00.121681 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebfc4926-f9e9-47c0-8d60-3874ddacab25 None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:00.122312 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebfc4926-f9e9-47c0-8d60-3874ddacab25 None admin] RBAC: Adding query filter params (type=image) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:24:00.122620 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebfc4926-f9e9-47c0-8d60-3874ddacab25 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:00.124680 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebfc4926-f9e9-47c0-8d60-3874ddacab25 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:00.134427 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 520/1044] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:23:59 2022] GET /identity/v3/services?type=image => generated 350 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:24:00.135648 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebfc4926-f9e9-47c0-8d60-3874ddacab25 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:00.148741 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8de25ea8-f69d-4324-953a-3a4a40d481be None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:00.188950 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13d87c5c-0168-4ff2-8360-c79b46126f59 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:00.189563 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13d87c5c-0168-4ff2-8360-c79b46126f59 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:00.189842 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13d87c5c-0168-4ff2-8360-c79b46126f59 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:00.190109 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13d87c5c-0168-4ff2-8360-c79b46126f59 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:00.209428 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:24:00.253980 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebfc4926-f9e9-47c0-8d60-3874ddacab25 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:00.265578 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13d87c5c-0168-4ff2-8360-c79b46126f59 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:00.267063 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13d87c5c-0168-4ff2-8360-c79b46126f59 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:00.288608 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3cfa1fb-2e35-4cf1-b3a1-31f4b4f71e6e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:00.289264 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3cfa1fb-2e35-4cf1-b3a1-31f4b4f71e6e None admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:00.289550 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3cfa1fb-2e35-4cf1-b3a1-31f4b4f71e6e None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:00.289823 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3cfa1fb-2e35-4cf1-b3a1-31f4b4f71e6e None admin] PATH_INFO: `/v3/endpoints` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:00.290487 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3cfa1fb-2e35-4cf1-b3a1-31f4b4f71e6e None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:00.291843 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3cfa1fb-2e35-4cf1-b3a1-31f4b4f71e6e None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:00.328672 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 521/1045] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:24:00 2022] POST /identity/v3/endpoints => generated 327 bytes in 182 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:24:00.328672 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7d9d6fe-872b-4c75-a6bb-cbdc3a263630 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:00.328672 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7d9d6fe-872b-4c75-a6bb-cbdc3a263630 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:00.328672 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7d9d6fe-872b-4c75-a6bb-cbdc3a263630 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:00.328672 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 522/1046] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:24:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:24:00.328672 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-192bef57-eb71-40a4-903a-5a33f1a2d26b None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:00.328672 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-192bef57-eb71-40a4-903a-5a33f1a2d26b None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:00.329373 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-192bef57-eb71-40a4-903a-5a33f1a2d26b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:00.348401 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-192bef57-eb71-40a4-903a-5a33f1a2d26b None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:24:00.349705 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-192bef57-eb71-40a4-903a-5a33f1a2d26b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:24:00.376915 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 525/1047] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Sep 25 04:24:00 2022] GET /identity/v3/auth/tokens => generated 1969 bytes in 359 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Sep 25 04:24:00.380968 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:24:00.382649 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13d87c5c-0168-4ff2-8360-c79b46126f59 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:00.438333 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 523/1048] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:24:00 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 25 04:24:00.449305 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:24:00.500568 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13d87c5c-0168-4ff2-8360-c79b46126f59 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:00.536151 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a794abf8-a3bd-4f5b-b763-416f2536f79f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:00.537210 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a794abf8-a3bd-4f5b-b763-416f2536f79f None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:00.537538 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a794abf8-a3bd-4f5b-b763-416f2536f79f None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:00.537810 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a794abf8-a3bd-4f5b-b763-416f2536f79f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:00.548687 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a794abf8-a3bd-4f5b-b763-416f2536f79f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:00.550313 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a794abf8-a3bd-4f5b-b763-416f2536f79f None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:00.566100 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-a794abf8-a3bd-4f5b-b763-416f2536f79f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:24:00.566100 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:24:00.566100 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:24:00.566100 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:24:00.566100 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:24:00.566100 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:24:00.566100 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:24:00.566100 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:24:00.566100 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:24:00.566100 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:24:00.566100 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:24:00.566100 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:24:00.566100 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 04:24:00.566100 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 04:24:00.566100 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 04:24:00.566100 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 04:24:00.567368 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:24:00.567368 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:24:00.567368 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:24:00.567368 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:24:00.567368 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:24:00.567368 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:24:00.567368 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:24:00.567368 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:24:00.567368 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:24:00.567368 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:24:00.567368 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:24:00.567368 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:24:00.567368 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:24:00.567368 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:24:00.567368 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:24:00.567368 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:24:00.567368 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 04:24:00.568510 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 04:24:00.568510 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:24:00.568510 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:24:00.570185 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 526/1049] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:24:00 2022] GET /identity/v3/domains/Default => generated 87 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 04:24:00.575830 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-192bef57-eb71-40a4-903a-5a33f1a2d26b None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:00.677542 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-192bef57-eb71-40a4-903a-5a33f1a2d26b None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:00.713843 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a794abf8-a3bd-4f5b-b763-416f2536f79f None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:00.720160 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2307347-13a3-49c0-acc6-660556ae7557 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:00.720882 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2307347-13a3-49c0-acc6-660556ae7557 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:00.721163 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2307347-13a3-49c0-acc6-660556ae7557 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:00.721431 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2307347-13a3-49c0-acc6-660556ae7557 None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:00.722067 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2307347-13a3-49c0-acc6-660556ae7557 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:24:00.722368 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2307347-13a3-49c0-acc6-660556ae7557 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:00.724159 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2307347-13a3-49c0-acc6-660556ae7557 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:00.737808 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 524/1050] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Sep 25 04:24:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:24:00.743237 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2307347-13a3-49c0-acc6-660556ae7557 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:00.765567 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:24:00.819222 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:24:00.823642 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a794abf8-a3bd-4f5b-b763-416f2536f79f None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:00.846165 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2307347-13a3-49c0-acc6-660556ae7557 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:00.860940 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcf86040-58f4-4d74-85d1-6c04c226bfa2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:00.861597 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf86040-58f4-4d74-85d1-6c04c226bfa2 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:00.861872 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf86040-58f4-4d74-85d1-6c04c226bfa2 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:00.862136 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf86040-58f4-4d74-85d1-6c04c226bfa2 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:00.873840 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcf86040-58f4-4d74-85d1-6c04c226bfa2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 04:24:00.878953 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-bcf86040-58f4-4d74-85d1-6c04c226bfa2 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 04:24:00.878953 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:24:00.878953 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:24:00.878953 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:24:00.878953 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:24:00.878953 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:24:00.878953 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:24:00.878953 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:24:00.878953 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:24:00.878953 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:24:00.878953 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:24:00.878953 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:24:00.878953 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 04:24:00.878953 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 04:24:00.878953 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 04:24:00.878953 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 04:24:00.879931 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 04:24:00.879931 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 04:24:00.879931 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 04:24:00.879931 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 04:24:00.879931 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:24:00.879931 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:24:00.879931 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:24:00.879931 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:24:00.879931 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:24:00.879931 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:24:00.879931 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:24:00.879931 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:24:00.879931 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:24:00.879931 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:24:00.879931 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:24:00.879931 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:24:00.880865 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:24:00.880865 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:24:00.880865 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:24:00.880865 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:24:00.880865 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 04:24:00.880865 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 04:24:00.880865 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 04:24:00.880865 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 04:24:00.880865 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 04:24:00.880865 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 04:24:00.880865 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 04:24:00.880865 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:24:00.880865 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 527/1051] 10.222.0.25 () {56 vars in 1096 bytes} [Sun Sep 25 04:24:00 2022] GET /identity/v3/roles/admin => generated 83 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 04:24:00.883459 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4958484-e318-46bd-a119-5d122c4ac62a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:00.884094 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4958484-e318-46bd-a119-5d122c4ac62a None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:00.884374 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4958484-e318-46bd-a119-5d122c4ac62a None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:00.884671 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4958484-e318-46bd-a119-5d122c4ac62a None admin] PATH_INFO: `/v3/projects/service` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:00.885831 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcf86040-58f4-4d74-85d1-6c04c226bfa2 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:00.891765 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4958484-e318-46bd-a119-5d122c4ac62a None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:00.893252 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4958484-e318-46bd-a119-5d122c4ac62a None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:00.904502 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-c4958484-e318-46bd-a119-5d122c4ac62a None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 04:24:00.904502 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:24:00.904502 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:24:00.904502 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:24:00.904502 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:24:00.904502 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:24:00.904502 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:24:00.904502 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:24:00.904502 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:24:00.904502 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:24:00.904502 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:24:00.904502 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:24:00.904502 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 25 04:24:00.904502 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 25 04:24:00.904502 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 25 04:24:00.905586 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 25 04:24:00.905586 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:24:00.905586 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:24:00.905586 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:24:00.905586 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:24:00.905586 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:24:00.905586 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:24:00.905586 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:24:00.905586 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:24:00.905586 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:24:00.905586 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:24:00.905586 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:24:00.905586 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:24:00.905586 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:24:00.905586 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:24:00.905586 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:24:00.905586 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:24:00.906573 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 25 04:24:00.906573 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 25 04:24:00.906573 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 25 04:24:00.906573 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 25 04:24:00.906573 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 25 04:24:00.906573 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 25 04:24:00.906573 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 04:24:00.906573 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:24:00.906573 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 525/1052] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Sep 25 04:24:00 2022] GET /identity/v3/projects/service => generated 88 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 04:24:00.911399 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4958484-e318-46bd-a119-5d122c4ac62a None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:00.993031 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:24:00.993333 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcf86040-58f4-4d74-85d1-6c04c226bfa2 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:01.014128 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4958484-e318-46bd-a119-5d122c4ac62a None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:01.030966 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed018cdc-6e59-4bb6-8ee2-84aaa7604884 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:01.031584 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed018cdc-6e59-4bb6-8ee2-84aaa7604884 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:01.031858 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed018cdc-6e59-4bb6-8ee2-84aaa7604884 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:01.032124 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed018cdc-6e59-4bb6-8ee2-84aaa7604884 None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:01.032803 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed018cdc-6e59-4bb6-8ee2-84aaa7604884 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:24:01.033108 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed018cdc-6e59-4bb6-8ee2-84aaa7604884 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:01.034457 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed018cdc-6e59-4bb6-8ee2-84aaa7604884 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:01.044958 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 528/1053] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Sep 25 04:24:00 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:24:01.048976 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-901940fd-5ad9-4f3d-bb93-7c1c6fa1370f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:01.049592 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-901940fd-5ad9-4f3d-bb93-7c1c6fa1370f None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:01.049874 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-901940fd-5ad9-4f3d-bb93-7c1c6fa1370f None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:01.050330 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-901940fd-5ad9-4f3d-bb93-7c1c6fa1370f None admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:01.051020 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-901940fd-5ad9-4f3d-bb93-7c1c6fa1370f None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:24:01.051200 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed018cdc-6e59-4bb6-8ee2-84aaa7604884 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:01.051356 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-901940fd-5ad9-4f3d-bb93-7c1c6fa1370f None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:01.052774 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-901940fd-5ad9-4f3d-bb93-7c1c6fa1370f None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:01.070616 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 526/1054] 10.222.0.25 () {56 vars in 1148 bytes} [Sun Sep 25 04:24:00 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:24:01.072004 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-901940fd-5ad9-4f3d-bb93-7c1c6fa1370f None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:01.083383 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:24:01.165351 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed018cdc-6e59-4bb6-8ee2-84aaa7604884 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:01.175867 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-901940fd-5ad9-4f3d-bb93-7c1c6fa1370f None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:01.203578 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c5eb637-ccf8-4850-8d31-34d31004f072 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:01.204223 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c5eb637-ccf8-4850-8d31-34d31004f072 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:01.204498 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c5eb637-ccf8-4850-8d31-34d31004f072 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:01.204800 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c5eb637-ccf8-4850-8d31-34d31004f072 None admin] PATH_INFO: `/v3/users/2f8b766c94e1454991f759bf9f1a8699` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:01.218618 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cca45e8-eccc-41c8-9ecf-14b3f398b944 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:01.219238 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cca45e8-eccc-41c8-9ecf-14b3f398b944 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:01.219518 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cca45e8-eccc-41c8-9ecf-14b3f398b944 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:01.219790 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cca45e8-eccc-41c8-9ecf-14b3f398b944 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:01.220106 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c5eb637-ccf8-4850-8d31-34d31004f072 None admin] RBAC: Authorizing `identity:get_user(user_id=2f8b766c94e1454991f759bf9f1a8699)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:01.220517 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cca45e8-eccc-41c8-9ecf-14b3f398b944 None admin] RBAC: Adding query filter params (role.id=062e3575fa1d40c880fc87ceca94db4a, scope.project.id=056ca1b1a9974a7581550de0d9217823, user.id=e703b42c53974daaa608b0ccc3f018b1) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:24:01.220853 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cca45e8-eccc-41c8-9ecf-14b3f398b944 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:01.221594 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c5eb637-ccf8-4850-8d31-34d31004f072 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:01.222275 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cca45e8-eccc-41c8-9ecf-14b3f398b944 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:01.223421 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 529/1055] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Sep 25 04:24:01 2022] GET /identity/v3/users/2f8b766c94e1454991f759bf9f1a8699 => generated 285 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:24:01.229098 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c5eb637-ccf8-4850-8d31-34d31004f072 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:01.237180 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 527/1056] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Sep 25 04:24:01 2022] GET /identity/v3/role_assignments?user.id=e703b42c53974daaa608b0ccc3f018b1&scope.project.id=056ca1b1a9974a7581550de0d9217823&role.id=062e3575fa1d40c880fc87ceca94db4a => generated 259 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:24:01.319679 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:24:01.341347 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c5eb637-ccf8-4850-8d31-34d31004f072 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:01.378294 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ac7db16-369f-4ddc-ae60-160a51b9cd4c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:01.378919 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ac7db16-369f-4ddc-ae60-160a51b9cd4c None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:01.379197 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ac7db16-369f-4ddc-ae60-160a51b9cd4c None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:01.379465 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ac7db16-369f-4ddc-ae60-160a51b9cd4c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:01.451414 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ac7db16-369f-4ddc-ae60-160a51b9cd4c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:01.453300 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ac7db16-369f-4ddc-ae60-160a51b9cd4c None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:01.540077 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:24:01.560882 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 530/1057] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Sep 25 04:24:01 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 332 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Sep 25 04:24:01.566849 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cca45e8-eccc-41c8-9ecf-14b3f398b944 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:01.571270 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:24:01.672275 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cca45e8-eccc-41c8-9ecf-14b3f398b944 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:01.706351 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb9f462f-dc83-49b0-9947-31b942179b53 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:01.706986 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb9f462f-dc83-49b0-9947-31b942179b53 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:01.707266 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb9f462f-dc83-49b0-9947-31b942179b53 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:01.707538 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb9f462f-dc83-49b0-9947-31b942179b53 None admin] PATH_INFO: `/v3/projects/ad29867a50584d078445e79ff33e3030` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:01.708676 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb9f462f-dc83-49b0-9947-31b942179b53 None admin] RBAC: Authorizing `identity:get_project(project_id=ad29867a50584d078445e79ff33e3030)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:01.710131 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb9f462f-dc83-49b0-9947-31b942179b53 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:01.712067 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 528/1058] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Sep 25 04:24:01 2022] GET /identity/v3/projects/ad29867a50584d078445e79ff33e3030 => generated 311 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:24:01.718700 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ac7db16-369f-4ddc-ae60-160a51b9cd4c None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:01.818029 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ac7db16-369f-4ddc-ae60-160a51b9cd4c None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:01.851845 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7530911-dce3-4bdd-b28e-aa9e20d9b11a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:01.852388 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7530911-dce3-4bdd-b28e-aa9e20d9b11a None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:01.852691 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7530911-dce3-4bdd-b28e-aa9e20d9b11a None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:01.852969 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7530911-dce3-4bdd-b28e-aa9e20d9b11a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:01.853688 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7530911-dce3-4bdd-b28e-aa9e20d9b11a None admin] RBAC: Adding query filter params (role.id=c51fe7c5ad804040879acfb54269455f, scope.project.id=ad29867a50584d078445e79ff33e3030, user.id=2f8b766c94e1454991f759bf9f1a8699) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:24:01.853997 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7530911-dce3-4bdd-b28e-aa9e20d9b11a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:01.855410 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7530911-dce3-4bdd-b28e-aa9e20d9b11a None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:01.861032 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 531/1059] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Sep 25 04:24:01 2022] GET /identity/v3/role_assignments?user.id=2f8b766c94e1454991f759bf9f1a8699&scope.project.id=ad29867a50584d078445e79ff33e3030&role.id=c51fe7c5ad804040879acfb54269455f => generated 259 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:24:02.535938 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-579a541d-61b9-428f-8bac-c133ee36b91e None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:02.536282 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-579a541d-61b9-428f-8bac-c133ee36b91e None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:02.536496 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-579a541d-61b9-428f-8bac-c133ee36b91e None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:02.538059 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 529/1060] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:24:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:24:02.543241 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c31c16d2-4984-42a2-a647-c254344af00c None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:02.543697 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c31c16d2-4984-42a2-a647-c254344af00c None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:02.543777 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c31c16d2-4984-42a2-a647-c254344af00c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:02.562714 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-c31c16d2-4984-42a2-a647-c254344af00c None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:24:02.563914 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-c31c16d2-4984-42a2-a647-c254344af00c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:24:02.606785 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:24:02.636771 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 532/1061] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:24:02 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 04:24:02.901341 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-579a541d-61b9-428f-8bac-c133ee36b91e None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:02.973777 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:24:03.006876 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-579a541d-61b9-428f-8bac-c133ee36b91e None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:03.041295 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-786c1023-ed09-4bff-a644-d52436ed4968 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:03.041910 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-786c1023-ed09-4bff-a644-d52436ed4968 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:03.042193 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-786c1023-ed09-4bff-a644-d52436ed4968 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:03.042468 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-786c1023-ed09-4bff-a644-d52436ed4968 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:03.111201 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-786c1023-ed09-4bff-a644-d52436ed4968 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:03.112677 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-786c1023-ed09-4bff-a644-d52436ed4968 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:03.180715 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:24:03.216678 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 530/1062] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Sep 25 04:24:02 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 313 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 25 04:24:03.220500 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c31c16d2-4984-42a2-a647-c254344af00c None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:03.323285 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c31c16d2-4984-42a2-a647-c254344af00c None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:03.358090 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54040624-59bf-4bbc-9cc3-a776f76c41d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:03.358994 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54040624-59bf-4bbc-9cc3-a776f76c41d1 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:03.359491 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54040624-59bf-4bbc-9cc3-a776f76c41d1 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:03.359982 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54040624-59bf-4bbc-9cc3-a776f76c41d1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:03.370369 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54040624-59bf-4bbc-9cc3-a776f76c41d1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:03.372028 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54040624-59bf-4bbc-9cc3-a776f76c41d1 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:03.387600 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-54040624-59bf-4bbc-9cc3-a776f76c41d1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:24:03.387600 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:24:03.387600 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:24:03.387600 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:24:03.387600 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:24:03.387600 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:24:03.387600 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:24:03.387600 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:24:03.387600 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:24:03.387600 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:24:03.387600 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:24:03.387600 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:24:03.387600 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 04:24:03.387600 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 04:24:03.387600 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 04:24:03.387600 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 04:24:03.388657 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:24:03.388657 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:24:03.388657 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:24:03.388657 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:24:03.388657 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:24:03.388657 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:24:03.388657 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:24:03.388657 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:24:03.388657 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:24:03.388657 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:24:03.388657 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:24:03.388657 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:24:03.388657 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:24:03.388657 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:24:03.388657 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:24:03.388657 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:24:03.388657 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 04:24:03.394872 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 04:24:03.394872 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:24:03.394872 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:24:03.394872 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 533/1063] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:24:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 04:24:03.396567 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-786c1023-ed09-4bff-a644-d52436ed4968 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:03.407841 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e1e8218-a1c2-4943-b2d0-eac3ac9f144a None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:03.408123 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e1e8218-a1c2-4943-b2d0-eac3ac9f144a None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:03.408398 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e1e8218-a1c2-4943-b2d0-eac3ac9f144a None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:03.409749 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 534/1064] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:24:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 25 04:24:03.414654 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c5bee08-2364-43db-b4fc-057a2256d754 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:03.414931 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c5bee08-2364-43db-b4fc-057a2256d754 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:03.415201 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c5bee08-2364-43db-b4fc-057a2256d754 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:03.435044 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-4c5bee08-2364-43db-b4fc-057a2256d754 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:24:03.436467 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-4c5bee08-2364-43db-b4fc-057a2256d754 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:24:03.458941 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:24:03.512621 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 535/1065] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:24:03 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 25 04:24:03.530831 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-786c1023-ed09-4bff-a644-d52436ed4968 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:03.567205 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbdfedd0-4f7f-425f-b80d-16136ba1790f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:03.567810 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbdfedd0-4f7f-425f-b80d-16136ba1790f None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:03.568091 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbdfedd0-4f7f-425f-b80d-16136ba1790f None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:03.568363 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbdfedd0-4f7f-425f-b80d-16136ba1790f None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:03.569023 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbdfedd0-4f7f-425f-b80d-16136ba1790f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:24:03.569329 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbdfedd0-4f7f-425f-b80d-16136ba1790f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:03.570693 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbdfedd0-4f7f-425f-b80d-16136ba1790f None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:03.583177 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 531/1066] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Sep 25 04:24:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:24:03.785727 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c5bee08-2364-43db-b4fc-057a2256d754 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:03.897803 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c5bee08-2364-43db-b4fc-057a2256d754 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:03.935404 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31fd16f7-a7c4-41ea-bda4-ef3ff8aec3d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:03.936631 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31fd16f7-a7c4-41ea-bda4-ef3ff8aec3d5 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:03.936933 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31fd16f7-a7c4-41ea-bda4-ef3ff8aec3d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:03.937218 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31fd16f7-a7c4-41ea-bda4-ef3ff8aec3d5 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:03.947077 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31fd16f7-a7c4-41ea-bda4-ef3ff8aec3d5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 04:24:03.952169 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-31fd16f7-a7c4-41ea-bda4-ef3ff8aec3d5 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 25 04:24:03.952169 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:24:03.952169 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:24:03.952169 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:24:03.952169 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:24:03.952169 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:24:03.952169 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:24:03.952169 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:24:03.952169 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:24:03.952169 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:24:03.952169 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:24:03.952169 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:24:03.952169 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 04:24:03.952169 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 04:24:03.952169 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 04:24:03.952169 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 04:24:03.953399 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 04:24:03.953399 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 04:24:03.953399 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 04:24:03.953399 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 04:24:03.953399 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:24:03.953399 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:24:03.953399 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:24:03.953399 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:24:03.953399 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:24:03.953399 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:24:03.953399 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:24:03.953399 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:24:03.953399 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:24:03.953399 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:24:03.953399 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:24:03.953399 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:24:03.954356 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:24:03.954356 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:24:03.954356 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:24:03.954356 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:24:03.954356 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 04:24:03.954356 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 04:24:03.954356 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 04:24:03.954356 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 04:24:03.954356 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 04:24:03.954356 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 04:24:03.954356 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 25 04:24:03.954356 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:24:03.954356 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 536/1067] 10.222.0.25 () {56 vars in 1120 bytes} [Sun Sep 25 04:24:03 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:24:03.959284 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbdfedd0-4f7f-425f-b80d-16136ba1790f None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:03.990935 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:24:04.014686 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-210edf6a-bf84-4bdf-8412-f37af9d30ed6 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:04.014968 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-210edf6a-bf84-4bdf-8412-f37af9d30ed6 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:04.015241 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-210edf6a-bf84-4bdf-8412-f37af9d30ed6 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:04.017266 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 537/1068] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:24:04 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 25 04:24:04.022171 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5849e5d-46e7-485e-a26b-59490d691ffb None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:04.022457 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5849e5d-46e7-485e-a26b-59490d691ffb None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:04.022730 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5849e5d-46e7-485e-a26b-59490d691ffb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:04.041750 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-a5849e5d-46e7-485e-a26b-59490d691ffb None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:24:04.042867 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-a5849e5d-46e7-485e-a26b-59490d691ffb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:24:04.054134 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:24:04.061489 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbdfedd0-4f7f-425f-b80d-16136ba1790f None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:04.096297 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0533e2e-d82c-4df4-9bb2-b05f85c0471f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:04.096937 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0533e2e-d82c-4df4-9bb2-b05f85c0471f None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:04.097222 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0533e2e-d82c-4df4-9bb2-b05f85c0471f None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:04.097496 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0533e2e-d82c-4df4-9bb2-b05f85c0471f None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:04.098156 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0533e2e-d82c-4df4-9bb2-b05f85c0471f None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:24:04.098461 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0533e2e-d82c-4df4-9bb2-b05f85c0471f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:04.099831 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0533e2e-d82c-4df4-9bb2-b05f85c0471f None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:04.108596 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 532/1069] 10.222.0.25 () {56 vars in 1115 bytes} [Sun Sep 25 04:24:03 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:24:04.110202 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 538/1070] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:24:04 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 25 04:24:04.114423 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0533e2e-d82c-4df4-9bb2-b05f85c0471f None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:04.208204 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:24:04.215423 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0533e2e-d82c-4df4-9bb2-b05f85c0471f None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:04.247834 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cb3bf21-2c8c-45b2-9666-4e2b4bedcfd7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:04.249259 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cb3bf21-2c8c-45b2-9666-4e2b4bedcfd7 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:04.249543 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cb3bf21-2c8c-45b2-9666-4e2b4bedcfd7 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:04.249815 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cb3bf21-2c8c-45b2-9666-4e2b4bedcfd7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:04.259290 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cb3bf21-2c8c-45b2-9666-4e2b4bedcfd7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:04.260740 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cb3bf21-2c8c-45b2-9666-4e2b4bedcfd7 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:04.275381 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-2cb3bf21-2c8c-45b2-9666-4e2b4bedcfd7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:24:04.275381 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:24:04.275381 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:24:04.275381 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:24:04.275381 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:24:04.275381 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:24:04.275381 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:24:04.275381 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:24:04.275381 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:24:04.275381 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:24:04.275381 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:24:04.275381 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:24:04.275381 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 04:24:04.275381 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 04:24:04.275381 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 04:24:04.275381 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 04:24:04.276587 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:24:04.276587 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:24:04.276587 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:24:04.276587 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:24:04.276587 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:24:04.276587 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:24:04.276587 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:24:04.276587 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:24:04.276587 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:24:04.276587 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:24:04.276587 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:24:04.276587 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:24:04.276587 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:24:04.276587 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:24:04.276587 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:24:04.276587 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:24:04.276587 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 04:24:04.277694 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 04:24:04.277694 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:24:04.277694 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:24:04.277694 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 533/1071] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:24:04 2022] GET /identity/v3/domains/Default => generated 87 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:24:04.282303 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5849e5d-46e7-485e-a26b-59490d691ffb None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:04.378892 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5849e5d-46e7-485e-a26b-59490d691ffb None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:04.379064 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cb3bf21-2c8c-45b2-9666-4e2b4bedcfd7 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:04.412823 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de7bb6bb-6ecf-4b74-9bca-44cd80b58d02 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:04.413422 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de7bb6bb-6ecf-4b74-9bca-44cd80b58d02 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:04.413700 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de7bb6bb-6ecf-4b74-9bca-44cd80b58d02 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:04.413969 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de7bb6bb-6ecf-4b74-9bca-44cd80b58d02 None admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:04.414592 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de7bb6bb-6ecf-4b74-9bca-44cd80b58d02 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:24:04.414891 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de7bb6bb-6ecf-4b74-9bca-44cd80b58d02 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:04.416234 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de7bb6bb-6ecf-4b74-9bca-44cd80b58d02 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:04.427745 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 539/1072] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Sep 25 04:24:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:24:04.433396 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de7bb6bb-6ecf-4b74-9bca-44cd80b58d02 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:04.442713 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:24:04.479259 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cb3bf21-2c8c-45b2-9666-4e2b4bedcfd7 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:04.513064 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aa7a7f9-c735-42d0-82b3-ee3acfd768a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:04.513703 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aa7a7f9-c735-42d0-82b3-ee3acfd768a4 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:04.514485 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aa7a7f9-c735-42d0-82b3-ee3acfd768a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:04.514759 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aa7a7f9-c735-42d0-82b3-ee3acfd768a4 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:04.525298 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aa7a7f9-c735-42d0-82b3-ee3acfd768a4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 04:24:04.530621 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-6aa7a7f9-c735-42d0-82b3-ee3acfd768a4 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 04:24:04.530621 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:24:04.530621 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:24:04.530621 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:24:04.530621 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:24:04.530621 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:24:04.530621 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:24:04.530621 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:24:04.530621 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:24:04.530621 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:24:04.530621 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:24:04.530621 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:24:04.530621 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 04:24:04.530621 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 04:24:04.530621 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 04:24:04.530621 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 04:24:04.531636 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 04:24:04.531636 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 04:24:04.531636 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 04:24:04.531636 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 04:24:04.531636 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:24:04.531636 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:24:04.531636 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:24:04.531636 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:24:04.531636 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:24:04.531636 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:24:04.531636 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:24:04.531636 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:24:04.531636 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:24:04.531636 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:24:04.531636 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:24:04.531636 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:24:04.533123 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:24:04.533123 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:24:04.533123 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:24:04.533123 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:24:04.533123 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 04:24:04.533123 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 04:24:04.533123 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 04:24:04.533123 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 04:24:04.533123 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 04:24:04.533123 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 04:24:04.533123 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 04:24:04.533123 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:24:04.533123 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 534/1073] 10.222.0.25 () {56 vars in 1096 bytes} [Sun Sep 25 04:24:04 2022] GET /identity/v3/roles/admin => generated 83 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 04:24:04.537686 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aa7a7f9-c735-42d0-82b3-ee3acfd768a4 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:04.540740 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de7bb6bb-6ecf-4b74-9bca-44cd80b58d02 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:04.546511 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:24:04.574023 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66a99eb1-9634-4a4a-92f9-bc3bcfe0bbf8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:04.574647 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66a99eb1-9634-4a4a-92f9-bc3bcfe0bbf8 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:04.574923 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66a99eb1-9634-4a4a-92f9-bc3bcfe0bbf8 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:04.575191 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66a99eb1-9634-4a4a-92f9-bc3bcfe0bbf8 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:04.583543 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66a99eb1-9634-4a4a-92f9-bc3bcfe0bbf8 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:04.585001 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66a99eb1-9634-4a4a-92f9-bc3bcfe0bbf8 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:04.596891 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-66a99eb1-9634-4a4a-92f9-bc3bcfe0bbf8 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 25 04:24:04.596891 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:24:04.596891 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:24:04.596891 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:24:04.596891 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:24:04.596891 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:24:04.596891 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:24:04.596891 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:24:04.596891 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:24:04.596891 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:24:04.596891 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:24:04.596891 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:24:04.596891 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 25 04:24:04.596891 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 25 04:24:04.596891 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 25 04:24:04.596891 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 25 04:24:04.598092 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:24:04.598092 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:24:04.598092 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 25 04:24:04.598092 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 04:24:04.598092 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 25 04:24:04.598092 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 04:24:04.598092 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:24:04.598092 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:24:04.598092 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:24:04.598092 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:24:04.598092 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:24:04.598092 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:24:04.598092 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:24:04.598092 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:24:04.598092 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:24:04.598092 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:24:04.598092 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:24:04.598092 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:24:04.599318 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:24:04.599318 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:24:04.599318 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 25 04:24:04.599318 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 25 04:24:04.599318 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 25 04:24:04.599318 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 25 04:24:04.599318 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 25 04:24:04.599318 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 25 04:24:04.599318 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 25 04:24:04.599318 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:24:04.599318 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 540/1074] 10.222.0.25 () {56 vars in 1093 bytes} [Sun Sep 25 04:24:04 2022] GET /identity/v3/users/nova => generated 82 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:24:04.604020 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66a99eb1-9634-4a4a-92f9-bc3bcfe0bbf8 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:04.639880 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aa7a7f9-c735-42d0-82b3-ee3acfd768a4 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:04.667742 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:24:04.673049 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37a0d588-c041-432c-bbe5-d06705ca8498 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:04.673658 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37a0d588-c041-432c-bbe5-d06705ca8498 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:04.673933 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37a0d588-c041-432c-bbe5-d06705ca8498 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:04.674202 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37a0d588-c041-432c-bbe5-d06705ca8498 None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:04.674851 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37a0d588-c041-432c-bbe5-d06705ca8498 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:24:04.675150 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37a0d588-c041-432c-bbe5-d06705ca8498 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:04.676510 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37a0d588-c041-432c-bbe5-d06705ca8498 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:04.685040 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 535/1075] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Sep 25 04:24:04 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:24:04.691377 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37a0d588-c041-432c-bbe5-d06705ca8498 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:04.704717 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66a99eb1-9634-4a4a-92f9-bc3bcfe0bbf8 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:04.737962 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4df00869-5256-4577-ad9d-963ce6bd12ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:04.738573 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4df00869-5256-4577-ad9d-963ce6bd12ed None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:04.738858 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4df00869-5256-4577-ad9d-963ce6bd12ed None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:04.739127 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4df00869-5256-4577-ad9d-963ce6bd12ed None admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:04.739850 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4df00869-5256-4577-ad9d-963ce6bd12ed None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:24:04.740157 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4df00869-5256-4577-ad9d-963ce6bd12ed None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:04.741611 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4df00869-5256-4577-ad9d-963ce6bd12ed None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:04.755859 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:24:04.759659 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 541/1076] 10.222.0.25 () {56 vars in 1133 bytes} [Sun Sep 25 04:24:04 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:24:04.766091 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4df00869-5256-4577-ad9d-963ce6bd12ed None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:04.793901 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37a0d588-c041-432c-bbe5-d06705ca8498 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:04.834799 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d07946e7-0036-4552-a79b-8d1b588297ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:04.835438 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d07946e7-0036-4552-a79b-8d1b588297ca None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:04.835718 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d07946e7-0036-4552-a79b-8d1b588297ca None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:04.835992 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d07946e7-0036-4552-a79b-8d1b588297ca None admin] PATH_INFO: `/v3/users/2f8b766c94e1454991f759bf9f1a8699` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:04.837150 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d07946e7-0036-4552-a79b-8d1b588297ca None admin] RBAC: Authorizing `identity:get_user(user_id=2f8b766c94e1454991f759bf9f1a8699)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:04.838580 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d07946e7-0036-4552-a79b-8d1b588297ca None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:04.840158 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 536/1077] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Sep 25 04:24:04 2022] GET /identity/v3/users/2f8b766c94e1454991f759bf9f1a8699 => generated 285 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:24:04.845983 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d07946e7-0036-4552-a79b-8d1b588297ca None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:04.864306 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4df00869-5256-4577-ad9d-963ce6bd12ed None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:04.887569 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:24:04.899069 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-053366af-d21c-4606-a7e0-fa88d78dd064 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:04.899636 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-053366af-d21c-4606-a7e0-fa88d78dd064 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:04.899912 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-053366af-d21c-4606-a7e0-fa88d78dd064 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:04.900183 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-053366af-d21c-4606-a7e0-fa88d78dd064 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:04.910247 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-053366af-d21c-4606-a7e0-fa88d78dd064 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:04.911682 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-053366af-d21c-4606-a7e0-fa88d78dd064 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:04.925941 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-053366af-d21c-4606-a7e0-fa88d78dd064 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:24:04.925941 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:24:04.925941 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:24:04.925941 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:24:04.925941 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:24:04.925941 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:24:04.925941 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:24:04.925941 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:24:04.925941 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:24:04.925941 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:24:04.925941 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:24:04.925941 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:24:04.925941 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 04:24:04.925941 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 04:24:04.925941 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 04:24:04.925941 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 04:24:04.927105 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:24:04.927105 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:24:04.927105 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:24:04.927105 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:24:04.927105 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:24:04.927105 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:24:04.927105 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:24:04.927105 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:24:04.927105 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:24:04.927105 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:24:04.927105 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:24:04.927105 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:24:04.927105 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:24:04.927105 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:24:04.927105 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:24:04.927105 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:24:04.927105 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 04:24:04.928385 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 04:24:04.928385 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:24:04.928385 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:24:04.928385 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 542/1078] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:24:04 2022] GET /identity/v3/domains/Default => generated 87 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:24:04.932549 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-053366af-d21c-4606-a7e0-fa88d78dd064 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:04.946111 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d07946e7-0036-4552-a79b-8d1b588297ca None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:04.981354 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94d3fcbb-1eb2-4728-bc32-76bd44651b01 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:04.981986 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d3fcbb-1eb2-4728-bc32-76bd44651b01 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:04.982272 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d3fcbb-1eb2-4728-bc32-76bd44651b01 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:04.982616 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d3fcbb-1eb2-4728-bc32-76bd44651b01 None admin] PATH_INFO: `/v3/projects/ad29867a50584d078445e79ff33e3030` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:04.983825 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94d3fcbb-1eb2-4728-bc32-76bd44651b01 None admin] RBAC: Authorizing `identity:get_project(project_id=ad29867a50584d078445e79ff33e3030)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:04.985313 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94d3fcbb-1eb2-4728-bc32-76bd44651b01 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:04.986935 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 537/1079] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Sep 25 04:24:04 2022] GET /identity/v3/projects/ad29867a50584d078445e79ff33e3030 => generated 311 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:24:04.992671 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94d3fcbb-1eb2-4728-bc32-76bd44651b01 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:04.995988 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:24:05.030750 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-053366af-d21c-4606-a7e0-fa88d78dd064 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:05.064597 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65596368-55ed-45b0-bf89-9e0f3721f9c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:05.065250 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65596368-55ed-45b0-bf89-9e0f3721f9c6 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:05.065532 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65596368-55ed-45b0-bf89-9e0f3721f9c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:05.065806 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65596368-55ed-45b0-bf89-9e0f3721f9c6 None admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:05.066447 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65596368-55ed-45b0-bf89-9e0f3721f9c6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:24:05.066746 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65596368-55ed-45b0-bf89-9e0f3721f9c6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:05.068103 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65596368-55ed-45b0-bf89-9e0f3721f9c6 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:05.081127 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 543/1080] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Sep 25 04:24:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:24:05.086714 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65596368-55ed-45b0-bf89-9e0f3721f9c6 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:05.091763 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94d3fcbb-1eb2-4728-bc32-76bd44651b01 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:05.121373 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:24:05.126179 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f05677c0-3248-458e-8364-46797865b7b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:05.127259 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f05677c0-3248-458e-8364-46797865b7b3 None admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:05.127539 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f05677c0-3248-458e-8364-46797865b7b3 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:05.127811 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f05677c0-3248-458e-8364-46797865b7b3 None admin] PATH_INFO: `/v3/projects/ad29867a50584d078445e79ff33e3030/users/2f8b766c94e1454991f759bf9f1a8699/roles/c51fe7c5ad804040879acfb54269455f` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:05.130356 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f05677c0-3248-458e-8364-46797865b7b3 None admin] RBAC: Authorizing `identity:create_grant(project_id=ad29867a50584d078445e79ff33e3030, user_id=2f8b766c94e1454991f759bf9f1a8699, role_id=c51fe7c5ad804040879acfb54269455f)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:05.131968 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f05677c0-3248-458e-8364-46797865b7b3 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:05.145757 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 538/1081] 10.222.0.25 () {58 vars in 1439 bytes} [Sun Sep 25 04:24:04 2022] PUT /identity/v3/projects/ad29867a50584d078445e79ff33e3030/users/2f8b766c94e1454991f759bf9f1a8699/roles/c51fe7c5ad804040879acfb54269455f => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:24:05.190379 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65596368-55ed-45b0-bf89-9e0f3721f9c6 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:05.226508 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8ce695b-b4d3-445f-9224-f92f2c21ee40 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:05.227152 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ce695b-b4d3-445f-9224-f92f2c21ee40 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:05.227424 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ce695b-b4d3-445f-9224-f92f2c21ee40 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:05.227693 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ce695b-b4d3-445f-9224-f92f2c21ee40 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:05.235647 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8ce695b-b4d3-445f-9224-f92f2c21ee40 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:05.237212 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8ce695b-b4d3-445f-9224-f92f2c21ee40 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:05.249735 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-e8ce695b-b4d3-445f-9224-f92f2c21ee40 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 04:24:05.249735 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:24:05.249735 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:24:05.249735 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:24:05.249735 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:24:05.249735 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:24:05.249735 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:24:05.249735 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:24:05.249735 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:24:05.249735 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:24:05.249735 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:24:05.249735 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:24:05.249735 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 25 04:24:05.249735 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 25 04:24:05.249735 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 25 04:24:05.251047 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 25 04:24:05.251047 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:24:05.251047 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:24:05.251047 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:24:05.251047 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:24:05.251047 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:24:05.251047 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:24:05.251047 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:24:05.251047 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:24:05.251047 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:24:05.251047 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:24:05.251047 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:24:05.251047 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:24:05.251047 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:24:05.251047 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:24:05.251047 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:24:05.251047 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:24:05.252241 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 25 04:24:05.252241 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 25 04:24:05.252241 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 25 04:24:05.252241 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 25 04:24:05.252241 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 25 04:24:05.252241 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 25 04:24:05.252241 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 04:24:05.252241 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:24:05.252241 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 544/1082] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Sep 25 04:24:05 2022] GET /identity/v3/projects/service => generated 88 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 04:24:05.256830 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f05677c0-3248-458e-8364-46797865b7b3 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:05.324622 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:24:05.365330 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f05677c0-3248-458e-8364-46797865b7b3 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:05.398883 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3288975-df28-41fa-a973-b99020ce97d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:05.399499 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3288975-df28-41fa-a973-b99020ce97d3 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:05.399780 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3288975-df28-41fa-a973-b99020ce97d3 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:05.400054 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3288975-df28-41fa-a973-b99020ce97d3 None admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:05.400775 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3288975-df28-41fa-a973-b99020ce97d3 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:24:05.401086 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3288975-df28-41fa-a973-b99020ce97d3 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:05.402505 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3288975-df28-41fa-a973-b99020ce97d3 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:05.414628 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 539/1083] 10.222.0.25 () {56 vars in 1148 bytes} [Sun Sep 25 04:24:05 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:24:05.420959 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8ce695b-b4d3-445f-9224-f92f2c21ee40 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:05.514763 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:24:05.520890 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8ce695b-b4d3-445f-9224-f92f2c21ee40 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:05.553186 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3656665b-6ad8-4fc2-8ae8-61e800eccec3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:05.553833 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3656665b-6ad8-4fc2-8ae8-61e800eccec3 None admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:05.554107 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3656665b-6ad8-4fc2-8ae8-61e800eccec3 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:05.554376 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3656665b-6ad8-4fc2-8ae8-61e800eccec3 None admin] PATH_INFO: `/v3/projects/056ca1b1a9974a7581550de0d9217823/users/e703b42c53974daaa608b0ccc3f018b1/roles/062e3575fa1d40c880fc87ceca94db4a` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:05.556462 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3656665b-6ad8-4fc2-8ae8-61e800eccec3 None admin] RBAC: Authorizing `identity:create_grant(project_id=056ca1b1a9974a7581550de0d9217823, user_id=e703b42c53974daaa608b0ccc3f018b1, role_id=062e3575fa1d40c880fc87ceca94db4a)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:05.557907 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3656665b-6ad8-4fc2-8ae8-61e800eccec3 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:05.572042 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 545/1084] 10.222.0.25 () {58 vars in 1439 bytes} [Sun Sep 25 04:24:05 2022] PUT /identity/v3/projects/056ca1b1a9974a7581550de0d9217823/users/e703b42c53974daaa608b0ccc3f018b1/roles/062e3575fa1d40c880fc87ceca94db4a => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:24:07.305835 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42ea14d3-7947-41e9-b9da-225ffe3f3ceb None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:07.306400 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42ea14d3-7947-41e9-b9da-225ffe3f3ceb None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:07.306400 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42ea14d3-7947-41e9-b9da-225ffe3f3ceb None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:07.307725 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 540/1085] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:24:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:24:07.313764 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cafb11a-ac77-4b58-9f4a-a624cf71e682 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:07.314202 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cafb11a-ac77-4b58-9f4a-a624cf71e682 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:07.314331 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cafb11a-ac77-4b58-9f4a-a624cf71e682 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:07.332545 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-1cafb11a-ac77-4b58-9f4a-a624cf71e682 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:24:07.333759 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-1cafb11a-ac77-4b58-9f4a-a624cf71e682 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:24:07.398742 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 546/1086] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:24:07 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 04:24:07.672016 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42ea14d3-7947-41e9-b9da-225ffe3f3ceb None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:07.706649 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3417b850-f7f2-4136-b61b-83d7d64ecf52 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:07.706930 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3417b850-f7f2-4136-b61b-83d7d64ecf52 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:07.707203 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3417b850-f7f2-4136-b61b-83d7d64ecf52 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:07.708725 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 547/1087] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:24:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 25 04:24:07.713576 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f7f596-7707-499b-a489-3599cfc98ce1 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:07.714197 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f7f596-7707-499b-a489-3599cfc98ce1 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:07.714197 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f7f596-7707-499b-a489-3599cfc98ce1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:07.732103 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-82f7f596-7707-499b-a489-3599cfc98ce1 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:24:07.734013 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-82f7f596-7707-499b-a489-3599cfc98ce1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:24:07.774915 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42ea14d3-7947-41e9-b9da-225ffe3f3ceb None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:07.798881 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:24:07.801650 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 548/1088] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:24:07 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 25 04:24:07.810535 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28ff2dc7-0658-423e-9a7c-288e28fd594d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:07.811223 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28ff2dc7-0658-423e-9a7c-288e28fd594d None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:07.811494 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28ff2dc7-0658-423e-9a7c-288e28fd594d None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:07.811758 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28ff2dc7-0658-423e-9a7c-288e28fd594d None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:07.822089 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28ff2dc7-0658-423e-9a7c-288e28fd594d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 04:24:07.827124 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-28ff2dc7-0658-423e-9a7c-288e28fd594d None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 04:24:07.827124 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:24:07.827124 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:24:07.827124 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:24:07.827124 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:24:07.827124 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:24:07.827124 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:24:07.827124 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:24:07.827124 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:24:07.827124 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:24:07.827124 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:24:07.827124 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:24:07.827124 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 04:24:07.827124 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 04:24:07.827124 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 04:24:07.827124 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 04:24:07.828282 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 04:24:07.828282 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 04:24:07.828282 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 04:24:07.828282 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 04:24:07.828282 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:24:07.828282 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:24:07.828282 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:24:07.828282 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:24:07.828282 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:24:07.828282 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:24:07.828282 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:24:07.828282 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:24:07.828282 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:24:07.828282 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:24:07.828282 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:24:07.828282 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:24:07.829196 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:24:07.829196 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:24:07.829196 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:24:07.829196 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:24:07.829196 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 04:24:07.829196 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 04:24:07.829196 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 04:24:07.829196 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 04:24:07.829196 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 04:24:07.829196 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 04:24:07.829196 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 04:24:07.829196 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:24:07.829196 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 541/1089] 10.222.0.25 () {56 vars in 1096 bytes} [Sun Sep 25 04:24:07 2022] GET /identity/v3/roles/admin => generated 83 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 04:24:07.834136 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82f7f596-7707-499b-a489-3599cfc98ce1 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:07.852075 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:24:07.931242 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82f7f596-7707-499b-a489-3599cfc98ce1 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:07.963461 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-549eb873-8fac-4059-b872-6c7b54760371 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:07.964072 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-549eb873-8fac-4059-b872-6c7b54760371 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:07.964347 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-549eb873-8fac-4059-b872-6c7b54760371 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:07.964619 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-549eb873-8fac-4059-b872-6c7b54760371 None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:07.965298 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-549eb873-8fac-4059-b872-6c7b54760371 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:24:07.965601 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-549eb873-8fac-4059-b872-6c7b54760371 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:07.967466 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-549eb873-8fac-4059-b872-6c7b54760371 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:07.975182 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 549/1090] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Sep 25 04:24:07 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:24:07.981076 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28ff2dc7-0658-423e-9a7c-288e28fd594d None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:08.074475 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-549eb873-8fac-4059-b872-6c7b54760371 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:08.081225 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28ff2dc7-0658-423e-9a7c-288e28fd594d None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:08.117208 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ce4fb9a-06a7-431c-9a4b-95ee2b0e027c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:08.117838 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce4fb9a-06a7-431c-9a4b-95ee2b0e027c None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:08.118119 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce4fb9a-06a7-431c-9a4b-95ee2b0e027c None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:08.118393 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce4fb9a-06a7-431c-9a4b-95ee2b0e027c None admin] PATH_INFO: `/v3/users/2f8b766c94e1454991f759bf9f1a8699` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:08.119517 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ce4fb9a-06a7-431c-9a4b-95ee2b0e027c None admin] RBAC: Authorizing `identity:get_user(user_id=2f8b766c94e1454991f759bf9f1a8699)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:08.120978 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ce4fb9a-06a7-431c-9a4b-95ee2b0e027c None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:08.122601 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 542/1091] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Sep 25 04:24:07 2022] GET /identity/v3/users/2f8b766c94e1454991f759bf9f1a8699 => generated 285 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:24:08.129051 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ce4fb9a-06a7-431c-9a4b-95ee2b0e027c None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:08.155356 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:24:08.177763 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-549eb873-8fac-4059-b872-6c7b54760371 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:08.182398 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:24:08.212174 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92b2f3b3-501f-45b0-b233-9eb48cfcb7d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:08.212827 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b2f3b3-501f-45b0-b233-9eb48cfcb7d4 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:08.213103 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b2f3b3-501f-45b0-b233-9eb48cfcb7d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:08.213368 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b2f3b3-501f-45b0-b233-9eb48cfcb7d4 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:08.223869 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92b2f3b3-501f-45b0-b233-9eb48cfcb7d4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 04:24:08.229028 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-92b2f3b3-501f-45b0-b233-9eb48cfcb7d4 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 25 04:24:08.229028 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:24:08.229028 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:24:08.229028 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:24:08.229028 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:24:08.229028 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:24:08.229028 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:24:08.229028 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:24:08.229028 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:24:08.229028 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:24:08.229028 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:24:08.229028 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:24:08.229028 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 04:24:08.229028 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 04:24:08.229028 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 04:24:08.229028 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 04:24:08.230117 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 04:24:08.230117 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 04:24:08.230117 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 04:24:08.230117 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 04:24:08.230117 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:24:08.230117 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:24:08.230117 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:24:08.230117 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:24:08.230117 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:24:08.230117 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:24:08.230117 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:24:08.230117 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:24:08.230117 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:24:08.230117 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:24:08.230117 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:24:08.230117 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:24:08.231479 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:24:08.231479 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:24:08.231479 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:24:08.231479 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:24:08.231479 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 04:24:08.231479 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 04:24:08.231479 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 04:24:08.231479 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 04:24:08.231479 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 04:24:08.231479 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 04:24:08.231479 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 25 04:24:08.231479 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:24:08.231479 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 550/1092] 10.222.0.25 () {56 vars in 1120 bytes} [Sun Sep 25 04:24:08 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 04:24:08.232569 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ce4fb9a-06a7-431c-9a4b-95ee2b0e027c None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:08.236158 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92b2f3b3-501f-45b0-b233-9eb48cfcb7d4 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:08.267137 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f49c638d-e55d-4fa0-b753-d666550156b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:08.267744 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f49c638d-e55d-4fa0-b753-d666550156b6 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:08.268028 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f49c638d-e55d-4fa0-b753-d666550156b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:08.268302 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f49c638d-e55d-4fa0-b753-d666550156b6 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:08.334589 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92b2f3b3-501f-45b0-b233-9eb48cfcb7d4 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:08.336585 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f49c638d-e55d-4fa0-b753-d666550156b6 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:08.338076 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f49c638d-e55d-4fa0-b753-d666550156b6 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:08.370646 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-970b0f7b-ea3c-45c4-b00b-8418f6603d16 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:08.371255 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-970b0f7b-ea3c-45c4-b00b-8418f6603d16 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:08.371540 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-970b0f7b-ea3c-45c4-b00b-8418f6603d16 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:08.371818 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-970b0f7b-ea3c-45c4-b00b-8418f6603d16 None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:08.372479 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-970b0f7b-ea3c-45c4-b00b-8418f6603d16 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:24:08.372812 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-970b0f7b-ea3c-45c4-b00b-8418f6603d16 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:08.375051 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-970b0f7b-ea3c-45c4-b00b-8418f6603d16 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:08.380431 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:24:08.388884 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 551/1093] 10.222.0.25 () {56 vars in 1115 bytes} [Sun Sep 25 04:24:08 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:24:08.389905 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-970b0f7b-ea3c-45c4-b00b-8418f6603d16 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:08.394021 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:24:08.441946 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 543/1094] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Sep 25 04:24:08 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 313 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Sep 25 04:24:08.448033 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f49c638d-e55d-4fa0-b753-d666550156b6 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:08.487448 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-970b0f7b-ea3c-45c4-b00b-8418f6603d16 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:08.522369 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e8f8220-487e-4e15-ab31-168c45d2d314 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:08.523400 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e8f8220-487e-4e15-ab31-168c45d2d314 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:08.523692 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e8f8220-487e-4e15-ab31-168c45d2d314 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:08.523966 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e8f8220-487e-4e15-ab31-168c45d2d314 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:08.535106 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e8f8220-487e-4e15-ab31-168c45d2d314 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:08.536551 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e8f8220-487e-4e15-ab31-168c45d2d314 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:08.548437 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f49c638d-e55d-4fa0-b753-d666550156b6 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:08.552341 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-4e8f8220-487e-4e15-ab31-168c45d2d314 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:24:08.552341 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:24:08.552341 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:24:08.552341 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:24:08.552341 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:24:08.552341 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:24:08.552341 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:24:08.552341 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:24:08.552341 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:24:08.552341 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:24:08.552341 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:24:08.552341 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:24:08.552341 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 04:24:08.552341 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 04:24:08.552341 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 04:24:08.552341 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 04:24:08.553355 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:24:08.553355 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:24:08.553355 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:24:08.553355 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:24:08.553355 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:24:08.553355 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:24:08.553355 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:24:08.553355 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:24:08.553355 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:24:08.553355 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:24:08.553355 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:24:08.553355 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:24:08.553355 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:24:08.553355 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:24:08.553355 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:24:08.553355 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:24:08.553355 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 04:24:08.554283 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 04:24:08.554283 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:24:08.554283 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:24:08.554283 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 552/1095] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:24:08 2022] GET /identity/v3/domains/Default => generated 87 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:24:08.559273 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e8f8220-487e-4e15-ab31-168c45d2d314 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:08.583147 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27a3c3bd-64a8-4f01-9d93-4364c0f2bde0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:08.583782 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27a3c3bd-64a8-4f01-9d93-4364c0f2bde0 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:08.584065 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27a3c3bd-64a8-4f01-9d93-4364c0f2bde0 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:08.584337 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27a3c3bd-64a8-4f01-9d93-4364c0f2bde0 None admin] PATH_INFO: `/v3/projects/ad29867a50584d078445e79ff33e3030` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:08.585483 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27a3c3bd-64a8-4f01-9d93-4364c0f2bde0 None admin] RBAC: Authorizing `identity:get_project(project_id=ad29867a50584d078445e79ff33e3030)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:08.586913 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27a3c3bd-64a8-4f01-9d93-4364c0f2bde0 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:08.588879 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 544/1096] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Sep 25 04:24:08 2022] GET /identity/v3/projects/ad29867a50584d078445e79ff33e3030 => generated 311 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:24:08.595141 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27a3c3bd-64a8-4f01-9d93-4364c0f2bde0 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:08.611321 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:24:08.617712 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:24:08.660050 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e8f8220-487e-4e15-ab31-168c45d2d314 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:08.693289 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad9a7378-0382-41db-8106-0339d974ef8b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:08.693822 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad9a7378-0382-41db-8106-0339d974ef8b None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:08.694100 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad9a7378-0382-41db-8106-0339d974ef8b None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:08.694367 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad9a7378-0382-41db-8106-0339d974ef8b None admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:08.694999 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad9a7378-0382-41db-8106-0339d974ef8b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:24:08.695301 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad9a7378-0382-41db-8106-0339d974ef8b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:08.695545 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27a3c3bd-64a8-4f01-9d93-4364c0f2bde0 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:08.696862 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad9a7378-0382-41db-8106-0339d974ef8b None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:08.709134 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 553/1097] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Sep 25 04:24:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:24:08.714713 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad9a7378-0382-41db-8106-0339d974ef8b None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:08.730324 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cb103ba-2d25-46c4-b81d-b498f96ea39a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:08.730932 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cb103ba-2d25-46c4-b81d-b498f96ea39a None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:08.731212 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cb103ba-2d25-46c4-b81d-b498f96ea39a None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:08.731484 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cb103ba-2d25-46c4-b81d-b498f96ea39a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:08.732197 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cb103ba-2d25-46c4-b81d-b498f96ea39a None admin] RBAC: Adding query filter params (role.id=c51fe7c5ad804040879acfb54269455f, scope.project.id=ad29867a50584d078445e79ff33e3030, user.id=2f8b766c94e1454991f759bf9f1a8699) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:24:08.733100 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cb103ba-2d25-46c4-b81d-b498f96ea39a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:08.734522 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cb103ba-2d25-46c4-b81d-b498f96ea39a None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:08.740426 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 545/1098] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Sep 25 04:24:08 2022] GET /identity/v3/role_assignments?user.id=2f8b766c94e1454991f759bf9f1a8699&scope.project.id=ad29867a50584d078445e79ff33e3030&role.id=c51fe7c5ad804040879acfb54269455f => generated 608 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:24:08.820435 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad9a7378-0382-41db-8106-0339d974ef8b None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:08.835585 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:24:08.858368 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00f54e33-b247-4733-87be-0372f1a1a2d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:08.859024 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00f54e33-b247-4733-87be-0372f1a1a2d8 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:08.859303 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00f54e33-b247-4733-87be-0372f1a1a2d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:08.859574 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00f54e33-b247-4733-87be-0372f1a1a2d8 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:08.867625 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00f54e33-b247-4733-87be-0372f1a1a2d8 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:08.869116 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00f54e33-b247-4733-87be-0372f1a1a2d8 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:08.881915 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-00f54e33-b247-4733-87be-0372f1a1a2d8 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 25 04:24:08.881915 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:24:08.881915 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:24:08.881915 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:24:08.881915 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:24:08.881915 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:24:08.881915 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:24:08.881915 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:24:08.881915 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:24:08.881915 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:24:08.881915 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:24:08.881915 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:24:08.881915 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 25 04:24:08.881915 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 25 04:24:08.881915 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 25 04:24:08.881915 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 25 04:24:08.883002 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:24:08.883002 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:24:08.883002 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 25 04:24:08.883002 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 04:24:08.883002 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 25 04:24:08.883002 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 04:24:08.883002 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:24:08.883002 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:24:08.883002 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:24:08.883002 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:24:08.883002 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:24:08.883002 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:24:08.883002 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:24:08.883002 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:24:08.883002 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:24:08.883002 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:24:08.883002 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:24:08.883002 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:24:08.884001 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:24:08.884001 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:24:08.884001 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 25 04:24:08.884001 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 25 04:24:08.884001 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 25 04:24:08.884001 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 25 04:24:08.884001 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 25 04:24:08.884001 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 25 04:24:08.884001 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 25 04:24:08.884001 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:24:08.884001 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 554/1099] 10.222.0.25 () {56 vars in 1093 bytes} [Sun Sep 25 04:24:08 2022] GET /identity/v3/users/nova => generated 82 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:24:08.889688 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cb103ba-2d25-46c4-b81d-b498f96ea39a None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:08.977809 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:24:08.994491 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cb103ba-2d25-46c4-b81d-b498f96ea39a None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:09.027492 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7536df04-c6c1-4e2b-a8f1-5d68138fca1a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:09.028101 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7536df04-c6c1-4e2b-a8f1-5d68138fca1a None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:09.028379 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7536df04-c6c1-4e2b-a8f1-5d68138fca1a None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:09.028664 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7536df04-c6c1-4e2b-a8f1-5d68138fca1a None admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:09.030060 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7536df04-c6c1-4e2b-a8f1-5d68138fca1a None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:24:09.030503 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7536df04-c6c1-4e2b-a8f1-5d68138fca1a None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:09.031925 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7536df04-c6c1-4e2b-a8f1-5d68138fca1a None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:09.049522 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 546/1100] 10.222.0.25 () {56 vars in 1133 bytes} [Sun Sep 25 04:24:08 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:24:09.055147 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00f54e33-b247-4733-87be-0372f1a1a2d8 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:09.153677 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00f54e33-b247-4733-87be-0372f1a1a2d8 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:09.186774 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34824512-6ee0-47dd-b99f-7fdf5d61764b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:09.187378 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34824512-6ee0-47dd-b99f-7fdf5d61764b None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:09.187657 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34824512-6ee0-47dd-b99f-7fdf5d61764b None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:09.187928 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34824512-6ee0-47dd-b99f-7fdf5d61764b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:09.228112 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:24:09.259342 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34824512-6ee0-47dd-b99f-7fdf5d61764b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:09.260833 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34824512-6ee0-47dd-b99f-7fdf5d61764b None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:09.356956 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 555/1101] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Sep 25 04:24:09 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 302 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Sep 25 04:24:09.363278 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7536df04-c6c1-4e2b-a8f1-5d68138fca1a None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:09.460859 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7536df04-c6c1-4e2b-a8f1-5d68138fca1a None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:09.495060 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6ec61da-6cce-4d56-b0ab-f0a1423da410 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:09.495697 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6ec61da-6cce-4d56-b0ab-f0a1423da410 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:09.495980 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6ec61da-6cce-4d56-b0ab-f0a1423da410 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:09.496253 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6ec61da-6cce-4d56-b0ab-f0a1423da410 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:09.507112 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6ec61da-6cce-4d56-b0ab-f0a1423da410 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:09.508557 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6ec61da-6cce-4d56-b0ab-f0a1423da410 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:09.510280 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:24:09.525261 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-c6ec61da-6cce-4d56-b0ab-f0a1423da410 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:24:09.525261 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:24:09.525261 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:24:09.525261 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:24:09.525261 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:24:09.525261 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:24:09.525261 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:24:09.525261 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:24:09.525261 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:24:09.525261 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:24:09.525261 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:24:09.525261 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:24:09.525261 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 04:24:09.525261 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 04:24:09.525261 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 04:24:09.525261 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 04:24:09.526251 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:24:09.526251 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:24:09.526251 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:24:09.526251 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:24:09.526251 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:24:09.526251 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:24:09.526251 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:24:09.526251 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:24:09.526251 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:24:09.526251 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:24:09.526251 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:24:09.526251 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:24:09.526251 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:24:09.526251 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:24:09.526251 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:24:09.526251 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:24:09.526251 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 04:24:09.527239 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 04:24:09.527239 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:24:09.527239 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:24:09.527239 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 547/1102] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:24:09 2022] GET /identity/v3/domains/Default => generated 87 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:24:09.532294 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34824512-6ee0-47dd-b99f-7fdf5d61764b None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:09.599484 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:24:09.630172 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34824512-6ee0-47dd-b99f-7fdf5d61764b None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:09.664763 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b58b8f8-aa86-4a1c-bd5b-be6986472dff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:09.665381 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b58b8f8-aa86-4a1c-bd5b-be6986472dff None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:09.665659 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b58b8f8-aa86-4a1c-bd5b-be6986472dff None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:09.665933 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b58b8f8-aa86-4a1c-bd5b-be6986472dff None admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:09.666579 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b58b8f8-aa86-4a1c-bd5b-be6986472dff None admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:24:09.666883 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b58b8f8-aa86-4a1c-bd5b-be6986472dff None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:09.668238 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b58b8f8-aa86-4a1c-bd5b-be6986472dff None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:09.679818 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 556/1103] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Sep 25 04:24:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:24:09.686116 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6ec61da-6cce-4d56-b0ab-f0a1423da410 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:09.782342 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6ec61da-6cce-4d56-b0ab-f0a1423da410 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:09.814818 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-add27698-27a6-4fed-8191-6c5d7663ffc9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:09.815430 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add27698-27a6-4fed-8191-6c5d7663ffc9 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:09.815707 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add27698-27a6-4fed-8191-6c5d7663ffc9 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:09.815973 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add27698-27a6-4fed-8191-6c5d7663ffc9 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:09.822311 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-add27698-27a6-4fed-8191-6c5d7663ffc9 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:09.823736 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-add27698-27a6-4fed-8191-6c5d7663ffc9 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:09.834858 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-add27698-27a6-4fed-8191-6c5d7663ffc9 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 04:24:09.834858 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:24:09.834858 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:24:09.834858 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:24:09.834858 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:24:09.834858 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:24:09.834858 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:24:09.834858 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:24:09.834858 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:24:09.834858 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:24:09.834858 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:24:09.834858 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:24:09.834858 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 25 04:24:09.834858 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 25 04:24:09.834858 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 25 04:24:09.836016 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 25 04:24:09.836016 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:24:09.836016 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:24:09.836016 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:24:09.836016 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:24:09.836016 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:24:09.836016 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:24:09.836016 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:24:09.836016 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:24:09.836016 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:24:09.836016 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:24:09.836016 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:24:09.836016 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:24:09.836016 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:24:09.836016 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:24:09.836016 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:24:09.836016 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:24:09.836983 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 25 04:24:09.836983 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 25 04:24:09.836983 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 25 04:24:09.836983 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 25 04:24:09.836983 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 25 04:24:09.836983 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 25 04:24:09.836983 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 04:24:09.836983 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:24:09.836983 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 548/1104] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Sep 25 04:24:09 2022] GET /identity/v3/projects/service => generated 88 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 04:24:09.841759 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b58b8f8-aa86-4a1c-bd5b-be6986472dff None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:09.938362 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b58b8f8-aa86-4a1c-bd5b-be6986472dff None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:09.965511 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:24:09.971275 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-356f6856-c9c8-4ec3-bee9-ec9c296cb422 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:09.971876 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-356f6856-c9c8-4ec3-bee9-ec9c296cb422 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:09.972148 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-356f6856-c9c8-4ec3-bee9-ec9c296cb422 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:09.972415 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-356f6856-c9c8-4ec3-bee9-ec9c296cb422 None admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:09.973124 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-356f6856-c9c8-4ec3-bee9-ec9c296cb422 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:24:09.973430 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-356f6856-c9c8-4ec3-bee9-ec9c296cb422 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:09.974839 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-356f6856-c9c8-4ec3-bee9-ec9c296cb422 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:09.987686 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 557/1105] 10.222.0.25 () {56 vars in 1148 bytes} [Sun Sep 25 04:24:09 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:24:09.993789 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-add27698-27a6-4fed-8191-6c5d7663ffc9 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:10.048582 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:24:10.092552 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-add27698-27a6-4fed-8191-6c5d7663ffc9 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:10.125227 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83b1224e-573d-4c35-b76a-67c3e0f14b38 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:10.125825 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83b1224e-573d-4c35-b76a-67c3e0f14b38 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:10.126107 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83b1224e-573d-4c35-b76a-67c3e0f14b38 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:10.126378 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83b1224e-573d-4c35-b76a-67c3e0f14b38 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:10.127088 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83b1224e-573d-4c35-b76a-67c3e0f14b38 None admin] RBAC: Adding query filter params (role.id=062e3575fa1d40c880fc87ceca94db4a, scope.project.id=056ca1b1a9974a7581550de0d9217823, user.id=e703b42c53974daaa608b0ccc3f018b1) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:24:10.127397 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83b1224e-573d-4c35-b76a-67c3e0f14b38 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:10.128829 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83b1224e-573d-4c35-b76a-67c3e0f14b38 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:10.134412 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 549/1106] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Sep 25 04:24:09 2022] GET /identity/v3/role_assignments?user.id=e703b42c53974daaa608b0ccc3f018b1&scope.project.id=056ca1b1a9974a7581550de0d9217823&role.id=062e3575fa1d40c880fc87ceca94db4a => generated 608 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:24:10.954010 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4821f641-5240-4b3d-a7e8-372ba0cd5e98 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:10.954561 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4821f641-5240-4b3d-a7e8-372ba0cd5e98 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:10.954561 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4821f641-5240-4b3d-a7e8-372ba0cd5e98 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:10.955902 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 558/1107] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:24:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 25 04:24:10.961422 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71efe2b7-e6eb-4471-8b88-2b4798972689 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:10.961835 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71efe2b7-e6eb-4471-8b88-2b4798972689 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:10.961975 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71efe2b7-e6eb-4471-8b88-2b4798972689 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:10.981364 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-71efe2b7-e6eb-4471-8b88-2b4798972689 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:24:10.982638 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-71efe2b7-e6eb-4471-8b88-2b4798972689 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:24:11.048368 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 550/1108] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:24:10 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 04:24:11.319254 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4821f641-5240-4b3d-a7e8-372ba0cd5e98 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:11.416058 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4821f641-5240-4b3d-a7e8-372ba0cd5e98 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:11.448751 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70217433-d12d-44cc-a0fe-ef8604a01155 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:11.449372 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70217433-d12d-44cc-a0fe-ef8604a01155 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:11.449650 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70217433-d12d-44cc-a0fe-ef8604a01155 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:11.449916 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70217433-d12d-44cc-a0fe-ef8604a01155 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:11.451039 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70217433-d12d-44cc-a0fe-ef8604a01155 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:11.452819 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70217433-d12d-44cc-a0fe-ef8604a01155 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:11.454346 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 559/1109] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:24:11 2022] GET /identity/v3/domains/default => generated 199 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:24:11.459937 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71efe2b7-e6eb-4471-8b88-2b4798972689 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:11.508106 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:24:11.557677 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71efe2b7-e6eb-4471-8b88-2b4798972689 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:11.590110 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42f738ef-9a53-4909-971e-c49c1466de5f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:11.590644 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42f738ef-9a53-4909-971e-c49c1466de5f None admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:11.590920 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42f738ef-9a53-4909-971e-c49c1466de5f None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:11.591189 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42f738ef-9a53-4909-971e-c49c1466de5f None admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:11.592537 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42f738ef-9a53-4909-971e-c49c1466de5f None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:11.594147 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42f738ef-9a53-4909-971e-c49c1466de5f None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:11.615411 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 551/1110] 10.222.0.25 () {60 vars in 1132 bytes} [Sun Sep 25 04:24:11 2022] POST /identity/v3/users => generated 286 bytes in 156 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:24:13.782561 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a547d734-8620-4b84-a188-2466c96546b4 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:13.783064 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a547d734-8620-4b84-a188-2466c96546b4 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:13.783064 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a547d734-8620-4b84-a188-2466c96546b4 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:13.784358 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 560/1111] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:24:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:24:13.789719 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f49825fe-8eed-4393-9808-f4bc6ff0544c None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:13.790118 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f49825fe-8eed-4393-9808-f4bc6ff0544c None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:13.790278 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f49825fe-8eed-4393-9808-f4bc6ff0544c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:13.808647 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-f49825fe-8eed-4393-9808-f4bc6ff0544c None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:24:13.809846 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-f49825fe-8eed-4393-9808-f4bc6ff0544c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:24:13.873880 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 552/1112] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:24:13 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 04:24:14.143968 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a547d734-8620-4b84-a188-2466c96546b4 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:14.202866 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:24:14.242893 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a547d734-8620-4b84-a188-2466c96546b4 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:14.275294 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0db6c9a1-2809-4b1b-8742-486dde787933 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:14.275911 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db6c9a1-2809-4b1b-8742-486dde787933 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:14.276191 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db6c9a1-2809-4b1b-8742-486dde787933 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:14.276463 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db6c9a1-2809-4b1b-8742-486dde787933 None admin] PATH_INFO: `/v3/roles/02a8d263519b4ff1b511a8577293126c` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:14.277882 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0db6c9a1-2809-4b1b-8742-486dde787933 None admin] RBAC: Authorizing `identity:get_role(role_id=02a8d263519b4ff1b511a8577293126c)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:14.279272 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0db6c9a1-2809-4b1b-8742-486dde787933 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:14.280561 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 561/1113] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Sep 25 04:24:14 2022] GET /identity/v3/roles/02a8d263519b4ff1b511a8577293126c => generated 223 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:24:14.286270 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f49825fe-8eed-4393-9808-f4bc6ff0544c None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:14.319192 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:24:14.384823 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f49825fe-8eed-4393-9808-f4bc6ff0544c None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:14.418249 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-137230a4-16ce-4b8b-adc6-d1dcc1ead1e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:14.418807 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-137230a4-16ce-4b8b-adc6-d1dcc1ead1e7 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:14.419082 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-137230a4-16ce-4b8b-adc6-d1dcc1ead1e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:14.419350 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-137230a4-16ce-4b8b-adc6-d1dcc1ead1e7 None admin] PATH_INFO: `/v3/users/f669c6d7b0c546038e2fcdbd68f9a039` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:14.432093 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-137230a4-16ce-4b8b-adc6-d1dcc1ead1e7 None admin] RBAC: Authorizing `identity:get_user(user_id=f669c6d7b0c546038e2fcdbd68f9a039)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:14.433549 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-137230a4-16ce-4b8b-adc6-d1dcc1ead1e7 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:14.435127 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 553/1114] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Sep 25 04:24:14 2022] GET /identity/v3/users/f669c6d7b0c546038e2fcdbd68f9a039 => generated 286 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:24:14.441540 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0db6c9a1-2809-4b1b-8742-486dde787933 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:14.538321 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0db6c9a1-2809-4b1b-8742-486dde787933 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:14.563891 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:24:14.571841 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e23ecf67-1415-4a2b-a70f-0b261d9d368d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:14.572439 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e23ecf67-1415-4a2b-a70f-0b261d9d368d None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:14.572740 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e23ecf67-1415-4a2b-a70f-0b261d9d368d None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:14.573007 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e23ecf67-1415-4a2b-a70f-0b261d9d368d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:14.637033 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e23ecf67-1415-4a2b-a70f-0b261d9d368d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:14.638505 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e23ecf67-1415-4a2b-a70f-0b261d9d368d None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:14.732059 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 562/1115] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Sep 25 04:24:14 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 291 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 25 04:24:14.738020 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-137230a4-16ce-4b8b-adc6-d1dcc1ead1e7 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:14.826480 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:24:14.836615 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-137230a4-16ce-4b8b-adc6-d1dcc1ead1e7 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:14.869769 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8472e4be-3c5f-4718-9106-169964ee462d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:14.870397 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8472e4be-3c5f-4718-9106-169964ee462d None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:14.870674 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8472e4be-3c5f-4718-9106-169964ee462d None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:14.870942 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8472e4be-3c5f-4718-9106-169964ee462d None admin] PATH_INFO: `/v3/projects/ad29867a50584d078445e79ff33e3030` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:14.872120 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8472e4be-3c5f-4718-9106-169964ee462d None admin] RBAC: Authorizing `identity:get_project(project_id=ad29867a50584d078445e79ff33e3030)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:14.874481 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8472e4be-3c5f-4718-9106-169964ee462d None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:14.876075 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 554/1116] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Sep 25 04:24:14 2022] GET /identity/v3/projects/ad29867a50584d078445e79ff33e3030 => generated 311 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:24:14.882311 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e23ecf67-1415-4a2b-a70f-0b261d9d368d None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:14.908718 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:24:14.978409 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e23ecf67-1415-4a2b-a70f-0b261d9d368d None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:15.010152 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2023c09b-69d7-4ed5-b15c-cbb5d2ed578a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:15.010693 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2023c09b-69d7-4ed5-b15c-cbb5d2ed578a None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:15.010972 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2023c09b-69d7-4ed5-b15c-cbb5d2ed578a None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:15.011240 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2023c09b-69d7-4ed5-b15c-cbb5d2ed578a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:15.011954 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2023c09b-69d7-4ed5-b15c-cbb5d2ed578a None admin] RBAC: Adding query filter params (role.id=02a8d263519b4ff1b511a8577293126c, scope.project.id=ad29867a50584d078445e79ff33e3030, user.id=f669c6d7b0c546038e2fcdbd68f9a039) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:24:15.012261 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2023c09b-69d7-4ed5-b15c-cbb5d2ed578a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:15.014407 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2023c09b-69d7-4ed5-b15c-cbb5d2ed578a None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:15.019543 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 563/1117] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Sep 25 04:24:14 2022] GET /identity/v3/role_assignments?user.id=f669c6d7b0c546038e2fcdbd68f9a039&scope.project.id=ad29867a50584d078445e79ff33e3030&role.id=02a8d263519b4ff1b511a8577293126c => generated 259 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:24:17.127137 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2425ae69-dab0-49dc-98f1-0e1deb858fd7 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:17.127707 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2425ae69-dab0-49dc-98f1-0e1deb858fd7 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:17.127707 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2425ae69-dab0-49dc-98f1-0e1deb858fd7 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:17.129057 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 555/1118] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:24:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:24:17.134196 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2c3d208-c29c-48c2-a51b-da97a61a811d None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:17.134599 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2c3d208-c29c-48c2-a51b-da97a61a811d None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:17.134741 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2c3d208-c29c-48c2-a51b-da97a61a811d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:17.153876 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-b2c3d208-c29c-48c2-a51b-da97a61a811d None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:24:17.155269 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-b2c3d208-c29c-48c2-a51b-da97a61a811d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:24:17.221515 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 564/1119] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:24:17 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 04:24:17.494879 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2425ae69-dab0-49dc-98f1-0e1deb858fd7 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:17.596851 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2425ae69-dab0-49dc-98f1-0e1deb858fd7 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:17.629964 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59e19e60-2c21-4f46-94b7-957c0dacdea0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:17.630587 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e19e60-2c21-4f46-94b7-957c0dacdea0 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:17.630872 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e19e60-2c21-4f46-94b7-957c0dacdea0 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:17.631148 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e19e60-2c21-4f46-94b7-957c0dacdea0 None admin] PATH_INFO: `/v3/roles/02a8d263519b4ff1b511a8577293126c` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:17.632542 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59e19e60-2c21-4f46-94b7-957c0dacdea0 None admin] RBAC: Authorizing `identity:get_role(role_id=02a8d263519b4ff1b511a8577293126c)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:17.633938 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59e19e60-2c21-4f46-94b7-957c0dacdea0 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:17.635257 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 556/1120] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Sep 25 04:24:17 2022] GET /identity/v3/roles/02a8d263519b4ff1b511a8577293126c => generated 223 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:24:17.641127 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2c3d208-c29c-48c2-a51b-da97a61a811d None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:17.664396 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:24:17.738600 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2c3d208-c29c-48c2-a51b-da97a61a811d None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:17.771393 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21bd1ca3-55da-440b-acfd-0150e6417f81 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:17.772007 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21bd1ca3-55da-440b-acfd-0150e6417f81 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:17.772282 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21bd1ca3-55da-440b-acfd-0150e6417f81 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:17.772549 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21bd1ca3-55da-440b-acfd-0150e6417f81 None admin] PATH_INFO: `/v3/users/f669c6d7b0c546038e2fcdbd68f9a039` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:17.773687 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21bd1ca3-55da-440b-acfd-0150e6417f81 None admin] RBAC: Authorizing `identity:get_user(user_id=f669c6d7b0c546038e2fcdbd68f9a039)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:17.775095 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21bd1ca3-55da-440b-acfd-0150e6417f81 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:17.776648 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 565/1121] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Sep 25 04:24:17 2022] GET /identity/v3/users/f669c6d7b0c546038e2fcdbd68f9a039 => generated 286 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:24:17.782514 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59e19e60-2c21-4f46-94b7-957c0dacdea0 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:17.810262 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:24:17.880915 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59e19e60-2c21-4f46-94b7-957c0dacdea0 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:17.914233 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeaa8b5c-8fbc-4056-a440-f47b4bda844d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:17.914850 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeaa8b5c-8fbc-4056-a440-f47b4bda844d None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:17.915131 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeaa8b5c-8fbc-4056-a440-f47b4bda844d None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:17.915404 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeaa8b5c-8fbc-4056-a440-f47b4bda844d None admin] PATH_INFO: `/v3/projects/ad29867a50584d078445e79ff33e3030` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:17.916512 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aeaa8b5c-8fbc-4056-a440-f47b4bda844d None admin] RBAC: Authorizing `identity:get_project(project_id=ad29867a50584d078445e79ff33e3030)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:17.917961 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aeaa8b5c-8fbc-4056-a440-f47b4bda844d None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:17.919564 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 557/1122] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Sep 25 04:24:17 2022] GET /identity/v3/projects/ad29867a50584d078445e79ff33e3030 => generated 311 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:24:17.925630 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21bd1ca3-55da-440b-acfd-0150e6417f81 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:18.011485 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:24:18.021700 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21bd1ca3-55da-440b-acfd-0150e6417f81 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:18.054382 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-283fe554-a154-4a53-946f-0e37fc7f3bee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:18.055024 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-283fe554-a154-4a53-946f-0e37fc7f3bee None admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:18.055301 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-283fe554-a154-4a53-946f-0e37fc7f3bee None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:18.055578 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-283fe554-a154-4a53-946f-0e37fc7f3bee None admin] PATH_INFO: `/v3/projects/ad29867a50584d078445e79ff33e3030/users/f669c6d7b0c546038e2fcdbd68f9a039/roles/02a8d263519b4ff1b511a8577293126c` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:18.057717 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-283fe554-a154-4a53-946f-0e37fc7f3bee None admin] RBAC: Authorizing `identity:create_grant(project_id=ad29867a50584d078445e79ff33e3030, user_id=f669c6d7b0c546038e2fcdbd68f9a039, role_id=02a8d263519b4ff1b511a8577293126c)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:18.059117 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-283fe554-a154-4a53-946f-0e37fc7f3bee None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:18.071439 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 566/1123] 10.222.0.25 () {58 vars in 1439 bytes} [Sun Sep 25 04:24:17 2022] PUT /identity/v3/projects/ad29867a50584d078445e79ff33e3030/users/f669c6d7b0c546038e2fcdbd68f9a039/roles/02a8d263519b4ff1b511a8577293126c => generated 0 bytes in 146 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:24:20.196533 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d7611b1-f46d-45ed-8d2f-1d344dea5684 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:20.197158 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d7611b1-f46d-45ed-8d2f-1d344dea5684 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:20.197252 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d7611b1-f46d-45ed-8d2f-1d344dea5684 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:20.198622 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 558/1124] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:24:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:24:20.204657 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90ef3534-187a-45fb-8445-133b3a72922e None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:20.205044 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90ef3534-187a-45fb-8445-133b3a72922e None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:20.205219 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90ef3534-187a-45fb-8445-133b3a72922e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:20.223740 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-90ef3534-187a-45fb-8445-133b3a72922e None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:24:20.224946 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-90ef3534-187a-45fb-8445-133b3a72922e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:24:20.288870 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 567/1125] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:24:20 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 04:24:20.554331 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d7611b1-f46d-45ed-8d2f-1d344dea5684 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:20.645374 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:24:20.652122 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d7611b1-f46d-45ed-8d2f-1d344dea5684 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:20.685726 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9e0f548-7022-48b5-aa38-0a9ac92e1ce0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:20.686341 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e0f548-7022-48b5-aa38-0a9ac92e1ce0 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:20.686622 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e0f548-7022-48b5-aa38-0a9ac92e1ce0 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:20.686897 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e0f548-7022-48b5-aa38-0a9ac92e1ce0 None admin] PATH_INFO: `/v3/roles/02a8d263519b4ff1b511a8577293126c` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:20.688281 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9e0f548-7022-48b5-aa38-0a9ac92e1ce0 None admin] RBAC: Authorizing `identity:get_role(role_id=02a8d263519b4ff1b511a8577293126c)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:20.689676 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9e0f548-7022-48b5-aa38-0a9ac92e1ce0 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:20.690979 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 559/1126] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Sep 25 04:24:20 2022] GET /identity/v3/roles/02a8d263519b4ff1b511a8577293126c => generated 223 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:24:20.696814 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90ef3534-187a-45fb-8445-133b3a72922e None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:20.774278 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:24:20.793539 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90ef3534-187a-45fb-8445-133b3a72922e None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:20.827100 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b10fc32-b004-489d-a8c4-e9d3cc501a2e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:20.827771 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b10fc32-b004-489d-a8c4-e9d3cc501a2e None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:20.828050 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b10fc32-b004-489d-a8c4-e9d3cc501a2e None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:20.828318 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b10fc32-b004-489d-a8c4-e9d3cc501a2e None admin] PATH_INFO: `/v3/users/f669c6d7b0c546038e2fcdbd68f9a039` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:20.829453 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b10fc32-b004-489d-a8c4-e9d3cc501a2e None admin] RBAC: Authorizing `identity:get_user(user_id=f669c6d7b0c546038e2fcdbd68f9a039)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:20.830866 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b10fc32-b004-489d-a8c4-e9d3cc501a2e None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:20.832406 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 568/1127] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Sep 25 04:24:20 2022] GET /identity/v3/users/f669c6d7b0c546038e2fcdbd68f9a039 => generated 286 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:24:20.838112 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9e0f548-7022-48b5-aa38-0a9ac92e1ce0 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:20.935166 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9e0f548-7022-48b5-aa38-0a9ac92e1ce0 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:20.969702 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27357d5f-71c6-4e98-9d6a-498320d5a2f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:20.970293 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27357d5f-71c6-4e98-9d6a-498320d5a2f5 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:20.970574 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27357d5f-71c6-4e98-9d6a-498320d5a2f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:20.970849 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27357d5f-71c6-4e98-9d6a-498320d5a2f5 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:20.999457 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:24:21.036131 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27357d5f-71c6-4e98-9d6a-498320d5a2f5 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:21.037529 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27357d5f-71c6-4e98-9d6a-498320d5a2f5 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:21.131743 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 560/1128] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Sep 25 04:24:20 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 294 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 25 04:24:21.137805 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b10fc32-b004-489d-a8c4-e9d3cc501a2e None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:21.234299 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b10fc32-b004-489d-a8c4-e9d3cc501a2e None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:21.266161 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-369cbf07-7981-4653-8bfe-430dd629ddb2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:21.266776 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-369cbf07-7981-4653-8bfe-430dd629ddb2 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:21.267060 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-369cbf07-7981-4653-8bfe-430dd629ddb2 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:21.267333 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-369cbf07-7981-4653-8bfe-430dd629ddb2 None admin] PATH_INFO: `/v3/projects/ad29867a50584d078445e79ff33e3030` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:21.268431 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-369cbf07-7981-4653-8bfe-430dd629ddb2 None admin] RBAC: Authorizing `identity:get_project(project_id=ad29867a50584d078445e79ff33e3030)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:21.269893 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-369cbf07-7981-4653-8bfe-430dd629ddb2 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:21.271487 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 569/1129] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Sep 25 04:24:21 2022] GET /identity/v3/projects/ad29867a50584d078445e79ff33e3030 => generated 311 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:24:21.277587 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27357d5f-71c6-4e98-9d6a-498320d5a2f5 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:21.337983 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:24:21.375407 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27357d5f-71c6-4e98-9d6a-498320d5a2f5 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:21.408292 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db0230b3-bbae-4c61-9ec5-ff12af668ee5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:21.408911 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db0230b3-bbae-4c61-9ec5-ff12af668ee5 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:21.409193 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db0230b3-bbae-4c61-9ec5-ff12af668ee5 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:21.409464 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db0230b3-bbae-4c61-9ec5-ff12af668ee5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:21.410175 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db0230b3-bbae-4c61-9ec5-ff12af668ee5 None admin] RBAC: Adding query filter params (role.id=02a8d263519b4ff1b511a8577293126c, scope.project.id=ad29867a50584d078445e79ff33e3030, user.id=f669c6d7b0c546038e2fcdbd68f9a039) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:24:21.410483 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db0230b3-bbae-4c61-9ec5-ff12af668ee5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:21.411898 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db0230b3-bbae-4c61-9ec5-ff12af668ee5 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:21.417519 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 561/1130] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Sep 25 04:24:21 2022] GET /identity/v3/role_assignments?user.id=f669c6d7b0c546038e2fcdbd68f9a039&scope.project.id=ad29867a50584d078445e79ff33e3030&role.id=02a8d263519b4ff1b511a8577293126c => generated 608 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:24:23.573296 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87969478-3820-40c0-a749-0b475572e189 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:23.573830 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87969478-3820-40c0-a749-0b475572e189 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:23.573830 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87969478-3820-40c0-a749-0b475572e189 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:23.575177 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 570/1131] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:24:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:24:23.580754 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86178bb2-f680-44e9-8ad2-27810c412f0c None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:23.581158 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86178bb2-f680-44e9-8ad2-27810c412f0c None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:23.581299 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86178bb2-f680-44e9-8ad2-27810c412f0c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:23.599737 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-86178bb2-f680-44e9-8ad2-27810c412f0c None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:24:23.600954 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-86178bb2-f680-44e9-8ad2-27810c412f0c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:24:23.664515 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 562/1132] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:24:23 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 04:24:23.933749 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87969478-3820-40c0-a749-0b475572e189 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:23.952404 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:24:24.033994 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87969478-3820-40c0-a749-0b475572e189 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:24.066260 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c895cfe7-2491-4cf0-a422-9cb802218590 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:24.066883 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c895cfe7-2491-4cf0-a422-9cb802218590 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:24.067165 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c895cfe7-2491-4cf0-a422-9cb802218590 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:24.067439 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c895cfe7-2491-4cf0-a422-9cb802218590 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:24.068550 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c895cfe7-2491-4cf0-a422-9cb802218590 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:24.070005 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c895cfe7-2491-4cf0-a422-9cb802218590 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:24.071540 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 571/1133] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:24:23 2022] GET /identity/v3/domains/default => generated 199 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:24:24.078177 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86178bb2-f680-44e9-8ad2-27810c412f0c None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:24.134099 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:24:24.178820 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86178bb2-f680-44e9-8ad2-27810c412f0c None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:24.211428 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6acdb1d3-0a49-4649-b8e6-3c2964320c11 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:24.211961 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6acdb1d3-0a49-4649-b8e6-3c2964320c11 None admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:24.212238 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6acdb1d3-0a49-4649-b8e6-3c2964320c11 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:24.212506 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6acdb1d3-0a49-4649-b8e6-3c2964320c11 None admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:24.213169 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6acdb1d3-0a49-4649-b8e6-3c2964320c11 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:24.214575 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6acdb1d3-0a49-4649-b8e6-3c2964320c11 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:24.240531 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 563/1134] 10.222.0.25 () {60 vars in 1141 bytes} [Sun Sep 25 04:24:24 2022] POST /identity/v3/projects => generated 311 bytes in 163 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:24:26.409863 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-744f0baa-3719-4ee0-88ad-3248b4e9e4d3 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:26.410399 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-744f0baa-3719-4ee0-88ad-3248b4e9e4d3 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:26.410399 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-744f0baa-3719-4ee0-88ad-3248b4e9e4d3 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:26.411712 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 572/1135] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:24:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:24:26.416948 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cb9dbe7-ecd2-47ad-8289-5cd95ba388eb None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:26.417386 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cb9dbe7-ecd2-47ad-8289-5cd95ba388eb None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:26.417494 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cb9dbe7-ecd2-47ad-8289-5cd95ba388eb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:26.436430 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-6cb9dbe7-ecd2-47ad-8289-5cd95ba388eb None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:24:26.437654 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-6cb9dbe7-ecd2-47ad-8289-5cd95ba388eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:24:26.503290 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 564/1136] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:24:26 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 04:24:26.781403 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-744f0baa-3719-4ee0-88ad-3248b4e9e4d3 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:26.863440 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:24:26.879980 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-744f0baa-3719-4ee0-88ad-3248b4e9e4d3 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:26.913564 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03f4618d-e6b5-46ac-b1aa-5b5b075cbd1c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:26.914178 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03f4618d-e6b5-46ac-b1aa-5b5b075cbd1c None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:26.914451 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03f4618d-e6b5-46ac-b1aa-5b5b075cbd1c None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:26.914712 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03f4618d-e6b5-46ac-b1aa-5b5b075cbd1c None admin] PATH_INFO: `/v3/domains/default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:26.915800 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03f4618d-e6b5-46ac-b1aa-5b5b075cbd1c None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:26.917244 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03f4618d-e6b5-46ac-b1aa-5b5b075cbd1c None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:26.918776 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 573/1137] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:24:26 2022] GET /identity/v3/domains/default => generated 199 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:24:26.924824 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cb9dbe7-ecd2-47ad-8289-5cd95ba388eb None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:26.965519 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:24:27.023029 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cb9dbe7-ecd2-47ad-8289-5cd95ba388eb None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:27.056105 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eef1b19a-8a28-44ca-b792-ab40609d2ea9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:27.056661 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef1b19a-8a28-44ca-b792-ab40609d2ea9 None admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:27.056944 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef1b19a-8a28-44ca-b792-ab40609d2ea9 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:27.057215 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef1b19a-8a28-44ca-b792-ab40609d2ea9 None admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:27.057867 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eef1b19a-8a28-44ca-b792-ab40609d2ea9 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:27.059290 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eef1b19a-8a28-44ca-b792-ab40609d2ea9 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:27.080312 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 565/1138] 10.222.0.25 () {60 vars in 1132 bytes} [Sun Sep 25 04:24:26 2022] POST /identity/v3/users => generated 286 bytes in 156 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:24:29.260104 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a34f698-7207-4ea3-b768-1ae63b7c6b61 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:29.260729 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a34f698-7207-4ea3-b768-1ae63b7c6b61 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:29.260729 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a34f698-7207-4ea3-b768-1ae63b7c6b61 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:29.262011 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 574/1139] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:24:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:24:29.267799 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e2f2a8-7d16-4c9e-ad45-088bf88629ea None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:29.268195 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e2f2a8-7d16-4c9e-ad45-088bf88629ea None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:29.268355 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e2f2a8-7d16-4c9e-ad45-088bf88629ea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:29.286053 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-f1e2f2a8-7d16-4c9e-ad45-088bf88629ea None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:24:29.287231 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-f1e2f2a8-7d16-4c9e-ad45-088bf88629ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:24:29.351569 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 566/1140] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:24:29 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 04:24:29.620482 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a34f698-7207-4ea3-b768-1ae63b7c6b61 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:29.718242 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a34f698-7207-4ea3-b768-1ae63b7c6b61 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:29.750552 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5346d9a-c8cd-4610-9ccb-db9ab7f5f7ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:29.751166 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5346d9a-c8cd-4610-9ccb-db9ab7f5f7ab None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:29.751443 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5346d9a-c8cd-4610-9ccb-db9ab7f5f7ab None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:29.751712 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5346d9a-c8cd-4610-9ccb-db9ab7f5f7ab None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:29.761404 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5346d9a-c8cd-4610-9ccb-db9ab7f5f7ab None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 04:24:29.766824 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-e5346d9a-c8cd-4610-9ccb-db9ab7f5f7ab None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 04:24:29.766824 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:24:29.766824 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:24:29.766824 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:24:29.766824 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:24:29.766824 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:24:29.766824 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:24:29.766824 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:24:29.766824 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:24:29.766824 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:24:29.766824 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:24:29.766824 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:24:29.766824 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 04:24:29.766824 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 04:24:29.766824 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 04:24:29.766824 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 04:24:29.767957 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 04:24:29.767957 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 04:24:29.767957 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 04:24:29.767957 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 04:24:29.767957 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:24:29.767957 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:24:29.767957 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:24:29.767957 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:24:29.767957 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:24:29.767957 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:24:29.767957 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:24:29.767957 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:24:29.767957 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:24:29.767957 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:24:29.767957 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:24:29.767957 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:24:29.769094 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:24:29.769094 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:24:29.769094 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:24:29.769094 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:24:29.769094 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 04:24:29.769094 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 04:24:29.769094 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 04:24:29.769094 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 04:24:29.769094 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 04:24:29.769094 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 04:24:29.769094 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 04:24:29.769094 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:24:29.769094 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 575/1141] 10.222.0.25 () {56 vars in 1096 bytes} [Sun Sep 25 04:24:29 2022] GET /identity/v3/roles/admin => generated 83 bytes in 148 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:24:29.773947 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1e2f2a8-7d16-4c9e-ad45-088bf88629ea None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:29.800106 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:24:29.871350 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1e2f2a8-7d16-4c9e-ad45-088bf88629ea None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:29.904196 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbe56fe3-ec02-4647-b14f-0e92be7884a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:29.904814 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe56fe3-ec02-4647-b14f-0e92be7884a6 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:29.905095 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe56fe3-ec02-4647-b14f-0e92be7884a6 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:29.905365 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe56fe3-ec02-4647-b14f-0e92be7884a6 None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:29.906015 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbe56fe3-ec02-4647-b14f-0e92be7884a6 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:24:29.906317 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbe56fe3-ec02-4647-b14f-0e92be7884a6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:29.907676 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbe56fe3-ec02-4647-b14f-0e92be7884a6 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:29.915375 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 567/1142] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Sep 25 04:24:29 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:24:29.921861 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5346d9a-c8cd-4610-9ccb-db9ab7f5f7ab None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:29.937253 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:24:30.024882 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5346d9a-c8cd-4610-9ccb-db9ab7f5f7ab None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:30.057183 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b688c25-4bb2-4a34-adc0-c8f3676ad2f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:30.057798 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b688c25-4bb2-4a34-adc0-c8f3676ad2f1 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:30.058074 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b688c25-4bb2-4a34-adc0-c8f3676ad2f1 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:30.058341 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b688c25-4bb2-4a34-adc0-c8f3676ad2f1 None admin] PATH_INFO: `/v3/users/37845934a2e749f987ab724fc387cfcc` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:30.071392 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b688c25-4bb2-4a34-adc0-c8f3676ad2f1 None admin] RBAC: Authorizing `identity:get_user(user_id=37845934a2e749f987ab724fc387cfcc)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:30.072847 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b688c25-4bb2-4a34-adc0-c8f3676ad2f1 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:30.074426 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 576/1143] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Sep 25 04:24:29 2022] GET /identity/v3/users/37845934a2e749f987ab724fc387cfcc => generated 286 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:24:30.080700 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbe56fe3-ec02-4647-b14f-0e92be7884a6 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:30.166573 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:24:30.179523 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbe56fe3-ec02-4647-b14f-0e92be7884a6 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:30.211431 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88ad68d4-1f43-437c-ae72-cf48f2ae6ba4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:30.212028 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88ad68d4-1f43-437c-ae72-cf48f2ae6ba4 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:30.212307 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88ad68d4-1f43-437c-ae72-cf48f2ae6ba4 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:30.212576 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88ad68d4-1f43-437c-ae72-cf48f2ae6ba4 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:30.278151 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88ad68d4-1f43-437c-ae72-cf48f2ae6ba4 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:30.280220 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88ad68d4-1f43-437c-ae72-cf48f2ae6ba4 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:30.362548 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:24:30.375258 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 568/1144] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Sep 25 04:24:30 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 295 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 25 04:24:30.381229 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b688c25-4bb2-4a34-adc0-c8f3676ad2f1 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:30.447918 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:24:30.478633 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b688c25-4bb2-4a34-adc0-c8f3676ad2f1 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:30.511355 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad93231f-cd88-4214-b4e3-331503ae6928 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:30.511976 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad93231f-cd88-4214-b4e3-331503ae6928 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:30.512254 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad93231f-cd88-4214-b4e3-331503ae6928 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:30.512525 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad93231f-cd88-4214-b4e3-331503ae6928 None admin] PATH_INFO: `/v3/projects/8b75291ef33e4da6a09ef3243d78a940` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:30.513896 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad93231f-cd88-4214-b4e3-331503ae6928 None admin] RBAC: Authorizing `identity:get_project(project_id=8b75291ef33e4da6a09ef3243d78a940)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:30.515280 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad93231f-cd88-4214-b4e3-331503ae6928 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:30.516910 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 577/1145] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Sep 25 04:24:30 2022] GET /identity/v3/projects/8b75291ef33e4da6a09ef3243d78a940 => generated 311 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:24:30.523078 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88ad68d4-1f43-437c-ae72-cf48f2ae6ba4 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:30.619758 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88ad68d4-1f43-437c-ae72-cf48f2ae6ba4 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:30.653551 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b5baead-38af-43d2-af10-9161c5688c38 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:30.654685 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b5baead-38af-43d2-af10-9161c5688c38 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:30.654963 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b5baead-38af-43d2-af10-9161c5688c38 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:30.655231 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b5baead-38af-43d2-af10-9161c5688c38 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:30.655937 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b5baead-38af-43d2-af10-9161c5688c38 None admin] RBAC: Adding query filter params (role.id=c51fe7c5ad804040879acfb54269455f, scope.project.id=8b75291ef33e4da6a09ef3243d78a940, user.id=37845934a2e749f987ab724fc387cfcc) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:24:30.656240 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b5baead-38af-43d2-af10-9161c5688c38 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:30.657673 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b5baead-38af-43d2-af10-9161c5688c38 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:30.662850 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 569/1146] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Sep 25 04:24:30 2022] GET /identity/v3/role_assignments?user.id=37845934a2e749f987ab724fc387cfcc&scope.project.id=8b75291ef33e4da6a09ef3243d78a940&role.id=c51fe7c5ad804040879acfb54269455f => generated 259 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:24:32.857251 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99767d80-8526-4ad9-9e19-ecffd2bf3871 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:32.857776 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99767d80-8526-4ad9-9e19-ecffd2bf3871 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:32.857776 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99767d80-8526-4ad9-9e19-ecffd2bf3871 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:32.859117 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 578/1147] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:24:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:24:32.865337 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35877e79-e1c6-4232-ba88-0fee335149de None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:32.865786 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35877e79-e1c6-4232-ba88-0fee335149de None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:32.865881 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35877e79-e1c6-4232-ba88-0fee335149de None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:32.885968 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-35877e79-e1c6-4232-ba88-0fee335149de None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:24:32.887182 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-35877e79-e1c6-4232-ba88-0fee335149de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:24:32.939907 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:24:32.960951 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 570/1148] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:24:32 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 04:24:33.227717 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99767d80-8526-4ad9-9e19-ecffd2bf3871 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:33.329025 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99767d80-8526-4ad9-9e19-ecffd2bf3871 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:33.362532 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95266bdd-f337-4c98-a30a-65ae462677b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:33.363150 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95266bdd-f337-4c98-a30a-65ae462677b7 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:33.363565 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95266bdd-f337-4c98-a30a-65ae462677b7 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:33.364037 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95266bdd-f337-4c98-a30a-65ae462677b7 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:33.370506 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:24:33.374730 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95266bdd-f337-4c98-a30a-65ae462677b7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 04:24:33.379639 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-95266bdd-f337-4c98-a30a-65ae462677b7 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 04:24:33.379639 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:24:33.379639 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:24:33.379639 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:24:33.379639 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:24:33.379639 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:24:33.379639 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:24:33.379639 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:24:33.379639 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:24:33.379639 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:24:33.379639 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:24:33.379639 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:24:33.379639 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 04:24:33.379639 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 04:24:33.379639 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 04:24:33.379639 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 04:24:33.380798 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 04:24:33.380798 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 04:24:33.380798 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 04:24:33.380798 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 04:24:33.380798 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:24:33.380798 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:24:33.380798 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:24:33.380798 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:24:33.380798 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:24:33.380798 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:24:33.380798 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:24:33.380798 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:24:33.380798 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:24:33.380798 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:24:33.380798 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:24:33.380798 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:24:33.381708 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:24:33.381708 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:24:33.381708 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:24:33.381708 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:24:33.381708 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 04:24:33.381708 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 04:24:33.381708 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 04:24:33.381708 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 04:24:33.381708 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 04:24:33.381708 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 04:24:33.381708 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 04:24:33.381708 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:24:33.381708 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 579/1149] 10.222.0.25 () {56 vars in 1096 bytes} [Sun Sep 25 04:24:33 2022] GET /identity/v3/roles/admin => generated 83 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:24:33.386799 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35877e79-e1c6-4232-ba88-0fee335149de None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:33.484474 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35877e79-e1c6-4232-ba88-0fee335149de None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:33.517294 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a742be61-466a-4309-8bc0-d9be408f33f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:33.517897 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a742be61-466a-4309-8bc0-d9be408f33f0 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:33.518179 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a742be61-466a-4309-8bc0-d9be408f33f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:33.518453 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a742be61-466a-4309-8bc0-d9be408f33f0 None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:33.519110 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a742be61-466a-4309-8bc0-d9be408f33f0 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:24:33.519419 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a742be61-466a-4309-8bc0-d9be408f33f0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:33.520815 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a742be61-466a-4309-8bc0-d9be408f33f0 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:33.528580 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 571/1150] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Sep 25 04:24:33 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:24:33.534831 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95266bdd-f337-4c98-a30a-65ae462677b7 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:33.630604 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95266bdd-f337-4c98-a30a-65ae462677b7 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:33.662898 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afad2526-5ab0-455c-a1a6-40bcae77b162 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:33.663504 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afad2526-5ab0-455c-a1a6-40bcae77b162 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:33.663780 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afad2526-5ab0-455c-a1a6-40bcae77b162 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:33.664047 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afad2526-5ab0-455c-a1a6-40bcae77b162 None admin] PATH_INFO: `/v3/users/37845934a2e749f987ab724fc387cfcc` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:33.665170 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afad2526-5ab0-455c-a1a6-40bcae77b162 None admin] RBAC: Authorizing `identity:get_user(user_id=37845934a2e749f987ab724fc387cfcc)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:33.666576 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afad2526-5ab0-455c-a1a6-40bcae77b162 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:33.668110 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 580/1151] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Sep 25 04:24:33 2022] GET /identity/v3/users/37845934a2e749f987ab724fc387cfcc => generated 286 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:24:33.674288 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a742be61-466a-4309-8bc0-d9be408f33f0 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:33.719042 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:24:33.771866 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a742be61-466a-4309-8bc0-d9be408f33f0 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:33.803910 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93647e49-947f-4ba9-bb34-1fc064c318d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:33.804528 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93647e49-947f-4ba9-bb34-1fc064c318d2 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:33.804884 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93647e49-947f-4ba9-bb34-1fc064c318d2 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:33.805159 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93647e49-947f-4ba9-bb34-1fc064c318d2 None admin] PATH_INFO: `/v3/projects/8b75291ef33e4da6a09ef3243d78a940` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:33.806275 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93647e49-947f-4ba9-bb34-1fc064c318d2 None admin] RBAC: Authorizing `identity:get_project(project_id=8b75291ef33e4da6a09ef3243d78a940)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:33.807696 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93647e49-947f-4ba9-bb34-1fc064c318d2 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:33.809325 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 572/1152] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Sep 25 04:24:33 2022] GET /identity/v3/projects/8b75291ef33e4da6a09ef3243d78a940 => generated 311 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:24:33.815262 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afad2526-5ab0-455c-a1a6-40bcae77b162 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:33.878700 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:24:33.913086 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afad2526-5ab0-455c-a1a6-40bcae77b162 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:33.945926 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d6c70ef-8522-4314-a893-469e47c489de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:33.946573 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d6c70ef-8522-4314-a893-469e47c489de None admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:33.946855 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d6c70ef-8522-4314-a893-469e47c489de None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:33.947129 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d6c70ef-8522-4314-a893-469e47c489de None admin] PATH_INFO: `/v3/projects/8b75291ef33e4da6a09ef3243d78a940/users/37845934a2e749f987ab724fc387cfcc/roles/c51fe7c5ad804040879acfb54269455f` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:33.949047 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d6c70ef-8522-4314-a893-469e47c489de None admin] RBAC: Authorizing `identity:create_grant(project_id=8b75291ef33e4da6a09ef3243d78a940, user_id=37845934a2e749f987ab724fc387cfcc, role_id=c51fe7c5ad804040879acfb54269455f)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:33.950459 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d6c70ef-8522-4314-a893-469e47c489de None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:33.963878 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 581/1153] 10.222.0.25 () {58 vars in 1439 bytes} [Sun Sep 25 04:24:33 2022] PUT /identity/v3/projects/8b75291ef33e4da6a09ef3243d78a940/users/37845934a2e749f987ab724fc387cfcc/roles/c51fe7c5ad804040879acfb54269455f => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:24:36.095661 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2677962a-2325-4395-b7ef-05a85deb7627 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:36.096382 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2677962a-2325-4395-b7ef-05a85deb7627 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:36.096382 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2677962a-2325-4395-b7ef-05a85deb7627 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:36.097619 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 573/1154] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:24:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:24:36.103048 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e0f3e87-73a5-4758-91a7-0cbfcec2eaae None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:36.103464 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e0f3e87-73a5-4758-91a7-0cbfcec2eaae None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:36.103596 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e0f3e87-73a5-4758-91a7-0cbfcec2eaae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:36.122092 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-4e0f3e87-73a5-4758-91a7-0cbfcec2eaae None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:24:36.123830 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-4e0f3e87-73a5-4758-91a7-0cbfcec2eaae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:24:36.188024 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 582/1155] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:24:36 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 04:24:36.457443 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2677962a-2325-4395-b7ef-05a85deb7627 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:36.557351 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2677962a-2325-4395-b7ef-05a85deb7627 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:36.569563 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:24:36.591954 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b0965f6-959c-42ae-81ed-f31675670394 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:36.592571 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b0965f6-959c-42ae-81ed-f31675670394 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:36.592879 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b0965f6-959c-42ae-81ed-f31675670394 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:36.593147 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b0965f6-959c-42ae-81ed-f31675670394 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:36.603337 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b0965f6-959c-42ae-81ed-f31675670394 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 04:24:36.608902 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-4b0965f6-959c-42ae-81ed-f31675670394 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 04:24:36.608902 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:24:36.608902 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:24:36.608902 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:24:36.608902 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:24:36.608902 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:24:36.608902 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:24:36.608902 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:24:36.608902 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:24:36.608902 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:24:36.608902 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:24:36.608902 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:24:36.608902 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 04:24:36.608902 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 04:24:36.608902 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 04:24:36.608902 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 04:24:36.610103 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 04:24:36.610103 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 04:24:36.610103 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 04:24:36.610103 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 04:24:36.610103 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:24:36.610103 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:24:36.610103 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:24:36.610103 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:24:36.610103 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:24:36.610103 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:24:36.610103 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:24:36.610103 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:24:36.610103 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:24:36.610103 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:24:36.610103 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:24:36.610103 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:24:36.611324 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:24:36.611324 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:24:36.611324 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:24:36.611324 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:24:36.611324 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 04:24:36.611324 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 04:24:36.611324 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 04:24:36.611324 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 04:24:36.611324 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 04:24:36.611324 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 04:24:36.611324 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 04:24:36.611324 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:24:36.611324 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 574/1156] 10.222.0.25 () {56 vars in 1096 bytes} [Sun Sep 25 04:24:36 2022] GET /identity/v3/roles/admin => generated 83 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:24:36.615980 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e0f3e87-73a5-4758-91a7-0cbfcec2eaae None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:36.668690 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:24:36.712025 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e0f3e87-73a5-4758-91a7-0cbfcec2eaae None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:36.744162 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85ba000a-df70-4a8a-8444-81b5af08f668 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:36.744781 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ba000a-df70-4a8a-8444-81b5af08f668 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:36.745056 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ba000a-df70-4a8a-8444-81b5af08f668 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:36.745321 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ba000a-df70-4a8a-8444-81b5af08f668 None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:36.745967 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85ba000a-df70-4a8a-8444-81b5af08f668 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:24:36.746264 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85ba000a-df70-4a8a-8444-81b5af08f668 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:36.747638 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85ba000a-df70-4a8a-8444-81b5af08f668 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:36.755229 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 583/1157] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Sep 25 04:24:36 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:24:36.760994 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b0965f6-959c-42ae-81ed-f31675670394 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:36.858170 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b0965f6-959c-42ae-81ed-f31675670394 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:36.890079 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5671f8bf-c45f-4a07-904a-4be9f250dc2a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:36.890685 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5671f8bf-c45f-4a07-904a-4be9f250dc2a None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:36.890962 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5671f8bf-c45f-4a07-904a-4be9f250dc2a None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:36.891234 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5671f8bf-c45f-4a07-904a-4be9f250dc2a None admin] PATH_INFO: `/v3/users/37845934a2e749f987ab724fc387cfcc` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:36.892325 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5671f8bf-c45f-4a07-904a-4be9f250dc2a None admin] RBAC: Authorizing `identity:get_user(user_id=37845934a2e749f987ab724fc387cfcc)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:36.893848 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5671f8bf-c45f-4a07-904a-4be9f250dc2a None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:36.895387 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 575/1158] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Sep 25 04:24:36 2022] GET /identity/v3/users/37845934a2e749f987ab724fc387cfcc => generated 286 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:24:36.901051 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85ba000a-df70-4a8a-8444-81b5af08f668 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:37.006302 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85ba000a-df70-4a8a-8444-81b5af08f668 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:37.022188 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:24:37.038937 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4126fcf7-2a98-499b-980c-b4b0be753f3e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:37.039537 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4126fcf7-2a98-499b-980c-b4b0be753f3e None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:37.039819 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4126fcf7-2a98-499b-980c-b4b0be753f3e None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:37.040091 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4126fcf7-2a98-499b-980c-b4b0be753f3e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:37.104749 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4126fcf7-2a98-499b-980c-b4b0be753f3e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:37.106186 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4126fcf7-2a98-499b-980c-b4b0be753f3e None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:37.199865 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 584/1159] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Sep 25 04:24:36 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 299 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 25 04:24:37.206113 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5671f8bf-c45f-4a07-904a-4be9f250dc2a None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:37.247946 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:24:37.304699 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5671f8bf-c45f-4a07-904a-4be9f250dc2a None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:37.337008 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8a31c39-76fc-4358-85b5-26d9e69a5615 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:37.337622 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8a31c39-76fc-4358-85b5-26d9e69a5615 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:37.337899 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8a31c39-76fc-4358-85b5-26d9e69a5615 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:37.338168 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8a31c39-76fc-4358-85b5-26d9e69a5615 None admin] PATH_INFO: `/v3/projects/8b75291ef33e4da6a09ef3243d78a940` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:37.339260 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8a31c39-76fc-4358-85b5-26d9e69a5615 None admin] RBAC: Authorizing `identity:get_project(project_id=8b75291ef33e4da6a09ef3243d78a940)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:37.340699 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8a31c39-76fc-4358-85b5-26d9e69a5615 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:37.342871 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 576/1160] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Sep 25 04:24:37 2022] GET /identity/v3/projects/8b75291ef33e4da6a09ef3243d78a940 => generated 311 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:24:37.348991 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4126fcf7-2a98-499b-980c-b4b0be753f3e None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:37.366688 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:24:37.452969 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4126fcf7-2a98-499b-980c-b4b0be753f3e None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:37.486529 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec88b384-0a19-4d5f-93b0-b1bb383d3f77 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:37.487137 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec88b384-0a19-4d5f-93b0-b1bb383d3f77 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:37.487418 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec88b384-0a19-4d5f-93b0-b1bb383d3f77 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:37.487691 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec88b384-0a19-4d5f-93b0-b1bb383d3f77 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:37.488409 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec88b384-0a19-4d5f-93b0-b1bb383d3f77 None admin] RBAC: Adding query filter params (role.id=c51fe7c5ad804040879acfb54269455f, scope.project.id=8b75291ef33e4da6a09ef3243d78a940, user.id=37845934a2e749f987ab724fc387cfcc) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:24:37.488754 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec88b384-0a19-4d5f-93b0-b1bb383d3f77 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:37.490175 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec88b384-0a19-4d5f-93b0-b1bb383d3f77 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:37.495753 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 585/1161] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Sep 25 04:24:37 2022] GET /identity/v3/role_assignments?user.id=37845934a2e749f987ab724fc387cfcc&scope.project.id=8b75291ef33e4da6a09ef3243d78a940&role.id=c51fe7c5ad804040879acfb54269455f => generated 608 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:24:39.644570 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ec5062f-08d8-4e95-a5a9-d9d044a3ac00 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:39.645148 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ec5062f-08d8-4e95-a5a9-d9d044a3ac00 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:39.645148 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ec5062f-08d8-4e95-a5a9-d9d044a3ac00 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:39.646465 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 577/1162] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:24:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:24:39.651676 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fa18c6a-3c38-4c9b-8abb-2ba3a6b63ec5 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:39.652102 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fa18c6a-3c38-4c9b-8abb-2ba3a6b63ec5 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:39.652196 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fa18c6a-3c38-4c9b-8abb-2ba3a6b63ec5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:39.670939 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-4fa18c6a-3c38-4c9b-8abb-2ba3a6b63ec5 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:24:39.672106 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-4fa18c6a-3c38-4c9b-8abb-2ba3a6b63ec5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:24:39.736226 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 586/1163] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:24:39 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 04:24:40.004624 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ec5062f-08d8-4e95-a5a9-d9d044a3ac00 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:40.103113 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ec5062f-08d8-4e95-a5a9-d9d044a3ac00 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:40.111738 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:24:40.136843 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d730109-e295-43cb-bb67-e8278ad131b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:40.137434 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d730109-e295-43cb-bb67-e8278ad131b1 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:40.137712 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d730109-e295-43cb-bb67-e8278ad131b1 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:40.138717 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d730109-e295-43cb-bb67-e8278ad131b1 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:40.202544 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d730109-e295-43cb-bb67-e8278ad131b1 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:40.203965 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d730109-e295-43cb-bb67-e8278ad131b1 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:40.297014 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 578/1164] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Sep 25 04:24:40 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 293 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 25 04:24:40.302843 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fa18c6a-3c38-4c9b-8abb-2ba3a6b63ec5 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:40.317407 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:24:40.399721 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fa18c6a-3c38-4c9b-8abb-2ba3a6b63ec5 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:40.432468 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbc66940-564a-487e-a7c9-d955bbef0f4b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:40.433105 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbc66940-564a-487e-a7c9-d955bbef0f4b None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:40.433382 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbc66940-564a-487e-a7c9-d955bbef0f4b None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:40.433648 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbc66940-564a-487e-a7c9-d955bbef0f4b None admin] PATH_INFO: `/v3/domains/swift_test` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:40.440314 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbc66940-564a-487e-a7c9-d955bbef0f4b None admin] RBAC: Authorizing `identity:get_domain(domain_id=swift_test)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:40.441770 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbc66940-564a-487e-a7c9-d955bbef0f4b None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:40.453815 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-dbc66940-564a-487e-a7c9-d955bbef0f4b None admin] Could not find domain: swift_test.: keystone.exception.DomainNotFound: Could not find domain: swift_test. Sep 25 04:24:40.453815 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:24:40.453815 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 712, in get_domain Sep 25 04:24:40.453815 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application project = self.driver.get_project(domain_id) Sep 25 04:24:40.453815 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 25 04:24:40.453815 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 25 04:24:40.453815 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 25 04:24:40.453815 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 25 04:24:40.453815 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: swift_test. Sep 25 04:24:40.453815 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:24:40.453815 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 25 04:24:40.453815 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:24:40.453815 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:24:40.453815 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:24:40.453815 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:24:40.453815 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:24:40.453815 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:24:40.455051 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:24:40.455051 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:24:40.455051 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:24:40.455051 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:24:40.455051 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:24:40.455051 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:24:40.455051 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 04:24:40.455051 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 04:24:40.455051 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 04:24:40.455051 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 04:24:40.455051 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:24:40.455051 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:24:40.455051 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:24:40.455051 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:24:40.455051 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:24:40.455051 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:24:40.456306 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:24:40.456306 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:24:40.456306 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:24:40.456306 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:24:40.456306 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:24:40.456306 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:24:40.456306 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:24:40.456306 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:24:40.456306 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:24:40.456306 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:24:40.456306 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 718, in get_domain Sep 25 04:24:40.456306 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 04:24:40.456306 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: swift_test. Sep 25 04:24:40.456306 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:24:40.456306 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 587/1165] 10.222.0.25 () {56 vars in 1117 bytes} [Sun Sep 25 04:24:40 2022] GET /identity/v3/domains/swift_test => generated 90 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:24:40.460818 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d730109-e295-43cb-bb67-e8278ad131b1 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:40.471579 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:24:40.562502 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d730109-e295-43cb-bb67-e8278ad131b1 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:40.596303 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58123cd9-e7c9-4d48-952a-7841d661082d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:40.596931 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58123cd9-e7c9-4d48-952a-7841d661082d None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:40.597215 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58123cd9-e7c9-4d48-952a-7841d661082d None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:40.597490 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58123cd9-e7c9-4d48-952a-7841d661082d None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:40.598123 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58123cd9-e7c9-4d48-952a-7841d661082d None admin] RBAC: Adding query filter params (name=swift_test) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:24:40.598428 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58123cd9-e7c9-4d48-952a-7841d661082d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:40.599792 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58123cd9-e7c9-4d48-952a-7841d661082d None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:40.607072 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 579/1166] 10.222.0.25 () {56 vars in 1115 bytes} [Sun Sep 25 04:24:40 2022] GET /identity/v3/domains?name=swift_test => generated 125 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:24:42.738859 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a8fb99d-0f36-46fe-b120-780f952b1c11 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:42.739381 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a8fb99d-0f36-46fe-b120-780f952b1c11 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:42.739381 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a8fb99d-0f36-46fe-b120-780f952b1c11 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:42.740660 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 588/1167] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:24:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:24:42.746064 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d0bc42-001a-40e2-87a2-bcabdc8a9f91 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:42.746467 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d0bc42-001a-40e2-87a2-bcabdc8a9f91 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:42.746623 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d0bc42-001a-40e2-87a2-bcabdc8a9f91 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:42.765166 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-a9d0bc42-001a-40e2-87a2-bcabdc8a9f91 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:24:42.766352 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-a9d0bc42-001a-40e2-87a2-bcabdc8a9f91 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:24:42.830844 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 580/1168] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:24:42 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 04:24:43.102161 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a8fb99d-0f36-46fe-b120-780f952b1c11 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:43.170385 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:24:43.199488 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a8fb99d-0f36-46fe-b120-780f952b1c11 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:43.233478 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eb1c656-6129-45f1-8962-f74d2a7f0dce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:43.234071 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eb1c656-6129-45f1-8962-f74d2a7f0dce None admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:43.234343 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eb1c656-6129-45f1-8962-f74d2a7f0dce None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:43.234607 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eb1c656-6129-45f1-8962-f74d2a7f0dce None admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:43.235268 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eb1c656-6129-45f1-8962-f74d2a7f0dce None admin] RBAC: Authorizing `identity:create_domain()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:43.236656 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eb1c656-6129-45f1-8962-f74d2a7f0dce None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:43.252096 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 589/1169] 10.222.0.25 () {60 vars in 1138 bytes} [Sun Sep 25 04:24:43 2022] POST /identity/v3/domains => generated 267 bytes in 150 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:24:45.411889 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f34a90-f679-4ee1-bff7-cfdb8a6674c8 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:45.412455 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f34a90-f679-4ee1-bff7-cfdb8a6674c8 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:45.412455 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f34a90-f679-4ee1-bff7-cfdb8a6674c8 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:45.413836 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 581/1170] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:24:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:24:45.419177 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb50ff02-9b36-4e28-9be1-eccf517db5b4 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:45.419623 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb50ff02-9b36-4e28-9be1-eccf517db5b4 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:45.419881 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb50ff02-9b36-4e28-9be1-eccf517db5b4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:45.438080 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-eb50ff02-9b36-4e28-9be1-eccf517db5b4 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:24:45.439283 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-eb50ff02-9b36-4e28-9be1-eccf517db5b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:24:45.506378 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 590/1171] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:24:45 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 04:24:45.780411 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8f34a90-f679-4ee1-bff7-cfdb8a6674c8 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:45.783285 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:24:45.880965 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8f34a90-f679-4ee1-bff7-cfdb8a6674c8 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:45.913794 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4f84c45-436f-4399-8c0e-0db0e42241e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:45.914412 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4f84c45-436f-4399-8c0e-0db0e42241e9 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:45.914694 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4f84c45-436f-4399-8c0e-0db0e42241e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:45.914968 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4f84c45-436f-4399-8c0e-0db0e42241e9 None admin] PATH_INFO: `/v3/domains/a954d5db70ab486394054c5f99e9f0fa` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:45.925514 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4f84c45-436f-4399-8c0e-0db0e42241e9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=a954d5db70ab486394054c5f99e9f0fa)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:45.926951 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4f84c45-436f-4399-8c0e-0db0e42241e9 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:45.928514 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 582/1172] 10.222.0.25 () {56 vars in 1183 bytes} [Sun Sep 25 04:24:45 2022] GET /identity/v3/domains/a954d5db70ab486394054c5f99e9f0fa => generated 267 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:24:45.934268 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb50ff02-9b36-4e28-9be1-eccf517db5b4 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:45.993250 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:24:46.031609 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb50ff02-9b36-4e28-9be1-eccf517db5b4 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:46.064104 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-847daded-7b8b-4ac9-9ac8-0598091220a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:46.064728 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-847daded-7b8b-4ac9-9ac8-0598091220a2 None admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:46.065012 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-847daded-7b8b-4ac9-9ac8-0598091220a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:46.065285 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-847daded-7b8b-4ac9-9ac8-0598091220a2 None admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:46.065944 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-847daded-7b8b-4ac9-9ac8-0598091220a2 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:46.067355 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-847daded-7b8b-4ac9-9ac8-0598091220a2 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:46.093118 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 591/1173] 10.222.0.25 () {60 vars in 1141 bytes} [Sun Sep 25 04:24:45 2022] POST /identity/v3/projects => generated 361 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 04:24:48.259103 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6628ee77-2ee4-45a0-a89a-93bb71ea8f99 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:48.259679 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6628ee77-2ee4-45a0-a89a-93bb71ea8f99 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:48.259679 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6628ee77-2ee4-45a0-a89a-93bb71ea8f99 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:48.261025 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 583/1174] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:24:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:24:48.266658 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d59e59b0-7697-48f9-b6d8-f0f08c53dc7f None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:48.267064 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d59e59b0-7697-48f9-b6d8-f0f08c53dc7f None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:48.267211 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d59e59b0-7697-48f9-b6d8-f0f08c53dc7f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:48.286286 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-d59e59b0-7697-48f9-b6d8-f0f08c53dc7f None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:24:48.287472 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-d59e59b0-7697-48f9-b6d8-f0f08c53dc7f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:24:48.351811 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 592/1175] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:24:48 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 04:24:48.620066 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6628ee77-2ee4-45a0-a89a-93bb71ea8f99 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:48.680304 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:24:48.720564 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6628ee77-2ee4-45a0-a89a-93bb71ea8f99 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:48.753347 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cf404fa-e490-45c4-8acd-5f513f5b3f39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:48.753968 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cf404fa-e490-45c4-8acd-5f513f5b3f39 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:48.754247 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cf404fa-e490-45c4-8acd-5f513f5b3f39 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:48.754517 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cf404fa-e490-45c4-8acd-5f513f5b3f39 None admin] PATH_INFO: `/v3/domains/a954d5db70ab486394054c5f99e9f0fa` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:48.755615 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cf404fa-e490-45c4-8acd-5f513f5b3f39 None admin] RBAC: Authorizing `identity:get_domain(domain_id=a954d5db70ab486394054c5f99e9f0fa)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:48.757066 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cf404fa-e490-45c4-8acd-5f513f5b3f39 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:48.758603 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 584/1176] 10.222.0.25 () {56 vars in 1183 bytes} [Sun Sep 25 04:24:48 2022] GET /identity/v3/domains/a954d5db70ab486394054c5f99e9f0fa => generated 267 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:24:48.764876 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d59e59b0-7697-48f9-b6d8-f0f08c53dc7f None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:48.807955 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:24:48.862019 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d59e59b0-7697-48f9-b6d8-f0f08c53dc7f None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:48.894485 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c97ee5a0-c7d1-4dbe-be31-314752c15608 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:48.895083 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c97ee5a0-c7d1-4dbe-be31-314752c15608 None admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:48.895362 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c97ee5a0-c7d1-4dbe-be31-314752c15608 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:48.895631 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c97ee5a0-c7d1-4dbe-be31-314752c15608 None admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:48.896279 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c97ee5a0-c7d1-4dbe-be31-314752c15608 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:48.897743 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c97ee5a0-c7d1-4dbe-be31-314752c15608 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:48.918058 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 593/1177] 10.222.0.25 () {60 vars in 1132 bytes} [Sun Sep 25 04:24:48 2022] POST /identity/v3/users => generated 311 bytes in 154 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:24:51.092261 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06925fc1-31ce-4571-939c-f65b79438301 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:51.092861 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06925fc1-31ce-4571-939c-f65b79438301 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:51.092861 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06925fc1-31ce-4571-939c-f65b79438301 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:51.094182 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 585/1178] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:24:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:24:51.099666 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae3e5e73-450d-420c-8811-cc440eaf999c None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:51.100116 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae3e5e73-450d-420c-8811-cc440eaf999c None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:51.100311 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae3e5e73-450d-420c-8811-cc440eaf999c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:51.118707 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-ae3e5e73-450d-420c-8811-cc440eaf999c None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:24:51.119880 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-ae3e5e73-450d-420c-8811-cc440eaf999c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:24:51.183864 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 594/1179] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:24:51 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 04:24:51.455653 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06925fc1-31ce-4571-939c-f65b79438301 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:51.555702 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06925fc1-31ce-4571-939c-f65b79438301 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:51.583160 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:24:51.588915 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e1a3b82-505a-489d-884d-3930ff764f4a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:51.589532 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e1a3b82-505a-489d-884d-3930ff764f4a None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:51.589807 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e1a3b82-505a-489d-884d-3930ff764f4a None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:51.590073 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e1a3b82-505a-489d-884d-3930ff764f4a None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:51.600741 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e1a3b82-505a-489d-884d-3930ff764f4a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 04:24:51.605471 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-2e1a3b82-505a-489d-884d-3930ff764f4a None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 04:24:51.605471 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:24:51.605471 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:24:51.605471 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:24:51.605471 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:24:51.605471 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:24:51.605471 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:24:51.605471 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:24:51.605471 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:24:51.605471 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:24:51.605471 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:24:51.605471 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:24:51.605471 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 04:24:51.605471 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 04:24:51.605471 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 04:24:51.605471 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 04:24:51.606779 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 04:24:51.606779 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 04:24:51.606779 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 04:24:51.606779 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 04:24:51.606779 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:24:51.606779 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:24:51.606779 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:24:51.606779 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:24:51.606779 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:24:51.606779 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:24:51.606779 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:24:51.606779 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:24:51.606779 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:24:51.606779 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:24:51.606779 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:24:51.606779 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:24:51.607969 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:24:51.607969 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:24:51.607969 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:24:51.607969 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:24:51.607969 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 04:24:51.607969 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 04:24:51.607969 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 04:24:51.607969 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 04:24:51.607969 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 04:24:51.607969 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 04:24:51.607969 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 04:24:51.607969 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:24:51.607969 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 586/1180] 10.222.0.25 () {56 vars in 1096 bytes} [Sun Sep 25 04:24:51 2022] GET /identity/v3/roles/admin => generated 83 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:24:51.612577 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae3e5e73-450d-420c-8811-cc440eaf999c None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:51.641247 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:24:51.708713 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae3e5e73-450d-420c-8811-cc440eaf999c None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:51.741874 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13cad39e-cc81-433c-8534-b5a728f3836c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:51.742468 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13cad39e-cc81-433c-8534-b5a728f3836c None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:51.742742 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13cad39e-cc81-433c-8534-b5a728f3836c None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:51.743009 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13cad39e-cc81-433c-8534-b5a728f3836c None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:51.743658 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13cad39e-cc81-433c-8534-b5a728f3836c None admin] RBAC: Adding query filter params (name=admin) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:24:51.743958 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13cad39e-cc81-433c-8534-b5a728f3836c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:51.745330 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13cad39e-cc81-433c-8534-b5a728f3836c None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:51.753140 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 595/1181] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Sep 25 04:24:51 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:24:51.758900 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e1a3b82-505a-489d-884d-3930ff764f4a None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:51.855320 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e1a3b82-505a-489d-884d-3930ff764f4a None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:51.887124 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2d07562-6a10-4572-8ce7-278321e1b3fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:51.888279 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2d07562-6a10-4572-8ce7-278321e1b3fc None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:51.888550 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2d07562-6a10-4572-8ce7-278321e1b3fc None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:51.888844 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2d07562-6a10-4572-8ce7-278321e1b3fc None admin] PATH_INFO: `/v3/users/d43efa97e9bb47289259bf298126bef0` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:51.901339 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2d07562-6a10-4572-8ce7-278321e1b3fc None admin] RBAC: Authorizing `identity:get_user(user_id=d43efa97e9bb47289259bf298126bef0)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:51.902757 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2d07562-6a10-4572-8ce7-278321e1b3fc None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:51.904394 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 587/1182] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Sep 25 04:24:51 2022] GET /identity/v3/users/d43efa97e9bb47289259bf298126bef0 => generated 311 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:24:51.910055 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13cad39e-cc81-433c-8534-b5a728f3836c None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:51.998413 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:24:52.007131 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13cad39e-cc81-433c-8534-b5a728f3836c None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:52.039376 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db3cb1b9-4633-4fbb-8fa9-12a4e3a30a30 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:52.039971 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db3cb1b9-4633-4fbb-8fa9-12a4e3a30a30 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:52.040245 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db3cb1b9-4633-4fbb-8fa9-12a4e3a30a30 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:52.040511 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db3cb1b9-4633-4fbb-8fa9-12a4e3a30a30 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:52.104920 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db3cb1b9-4633-4fbb-8fa9-12a4e3a30a30 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:52.106394 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db3cb1b9-4633-4fbb-8fa9-12a4e3a30a30 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:52.191416 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:24:52.200782 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 596/1183] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Sep 25 04:24:51 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 291 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 25 04:24:52.207035 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2d07562-6a10-4572-8ce7-278321e1b3fc None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:52.254751 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:24:52.303581 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2d07562-6a10-4572-8ce7-278321e1b3fc None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:52.336272 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea89e6c2-9d29-4d4a-80bb-7022b6500563 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:52.336910 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea89e6c2-9d29-4d4a-80bb-7022b6500563 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:52.337190 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea89e6c2-9d29-4d4a-80bb-7022b6500563 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:52.337459 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea89e6c2-9d29-4d4a-80bb-7022b6500563 None admin] PATH_INFO: `/v3/projects/da9e376f67414d7096edc245ae2e0b1c` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:52.338537 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea89e6c2-9d29-4d4a-80bb-7022b6500563 None admin] RBAC: Authorizing `identity:get_project(project_id=da9e376f67414d7096edc245ae2e0b1c)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:52.339959 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea89e6c2-9d29-4d4a-80bb-7022b6500563 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:52.341831 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 588/1184] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Sep 25 04:24:52 2022] GET /identity/v3/projects/da9e376f67414d7096edc245ae2e0b1c => generated 361 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:24:52.347878 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db3cb1b9-4633-4fbb-8fa9-12a4e3a30a30 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:52.442620 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db3cb1b9-4633-4fbb-8fa9-12a4e3a30a30 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:52.474832 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-816b5b73-402b-4d27-9d0c-b3e76293505f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:52.475440 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-816b5b73-402b-4d27-9d0c-b3e76293505f None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:52.475721 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-816b5b73-402b-4d27-9d0c-b3e76293505f None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:52.475992 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-816b5b73-402b-4d27-9d0c-b3e76293505f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:52.476746 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-816b5b73-402b-4d27-9d0c-b3e76293505f None admin] RBAC: Adding query filter params (role.id=c51fe7c5ad804040879acfb54269455f, scope.project.id=da9e376f67414d7096edc245ae2e0b1c, user.id=d43efa97e9bb47289259bf298126bef0) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:24:52.477056 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-816b5b73-402b-4d27-9d0c-b3e76293505f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:52.478462 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-816b5b73-402b-4d27-9d0c-b3e76293505f None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:52.484397 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 597/1185] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Sep 25 04:24:52 2022] GET /identity/v3/role_assignments?user.id=d43efa97e9bb47289259bf298126bef0&scope.project.id=da9e376f67414d7096edc245ae2e0b1c&role.id=c51fe7c5ad804040879acfb54269455f => generated 259 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:24:54.615668 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9ecd5f9-482f-4bf8-a002-461ec4d7b85e None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:54.616230 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9ecd5f9-482f-4bf8-a002-461ec4d7b85e None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:54.616230 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9ecd5f9-482f-4bf8-a002-461ec4d7b85e None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:54.617573 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 589/1186] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:24:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:24:54.624289 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50a22e6c-b37d-4b92-9fe3-b048a443390a None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:54.624728 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50a22e6c-b37d-4b92-9fe3-b048a443390a None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:54.624889 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50a22e6c-b37d-4b92-9fe3-b048a443390a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:54.644804 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-50a22e6c-b37d-4b92-9fe3-b048a443390a None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:24:54.645980 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-50a22e6c-b37d-4b92-9fe3-b048a443390a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:24:54.694469 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:24:54.710484 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 598/1187] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:24:54 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 25 04:24:54.985285 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9ecd5f9-482f-4bf8-a002-461ec4d7b85e None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:55.084471 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9ecd5f9-482f-4bf8-a002-461ec4d7b85e None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:55.102132 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:24:55.120836 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a11a7ef-471d-4903-b89b-c1f51f72a7cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:55.121486 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a11a7ef-471d-4903-b89b-c1f51f72a7cd None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:55.121761 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a11a7ef-471d-4903-b89b-c1f51f72a7cd None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:55.122254 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a11a7ef-471d-4903-b89b-c1f51f72a7cd None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:55.132298 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a11a7ef-471d-4903-b89b-c1f51f72a7cd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 04:24:55.137007 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-6a11a7ef-471d-4903-b89b-c1f51f72a7cd None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 04:24:55.137007 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:24:55.137007 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:24:55.137007 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:24:55.137007 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:24:55.137007 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:24:55.137007 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:24:55.137007 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:24:55.137007 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:24:55.137007 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:24:55.137007 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:24:55.137007 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:24:55.137007 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 04:24:55.137007 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 04:24:55.137007 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 04:24:55.137007 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 04:24:55.138485 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 04:24:55.138485 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 04:24:55.138485 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 04:24:55.138485 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 04:24:55.138485 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:24:55.138485 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:24:55.138485 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:24:55.138485 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:24:55.138485 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:24:55.138485 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:24:55.138485 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:24:55.138485 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:24:55.138485 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:24:55.138485 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:24:55.138485 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:24:55.138485 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:24:55.139353 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:24:55.139353 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:24:55.139353 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:24:55.139353 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:24:55.139353 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 04:24:55.139353 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 04:24:55.139353 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 04:24:55.139353 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 04:24:55.139353 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 04:24:55.139353 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 04:24:55.139353 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 04:24:55.139353 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:24:55.139353 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 590/1188] 10.222.0.25 () {56 vars in 1096 bytes} [Sun Sep 25 04:24:54 2022] GET /identity/v3/roles/admin => generated 83 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:24:55.144452 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50a22e6c-b37d-4b92-9fe3-b048a443390a None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:55.240512 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50a22e6c-b37d-4b92-9fe3-b048a443390a None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:55.272217 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c6ff340-362e-4518-b895-e0c5c153d1c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:55.272843 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c6ff340-362e-4518-b895-e0c5c153d1c5 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:55.273127 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c6ff340-362e-4518-b895-e0c5c153d1c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:55.273399 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c6ff340-362e-4518-b895-e0c5c153d1c5 None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:55.274054 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c6ff340-362e-4518-b895-e0c5c153d1c5 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:24:55.274358 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c6ff340-362e-4518-b895-e0c5c153d1c5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:55.275715 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c6ff340-362e-4518-b895-e0c5c153d1c5 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:55.283985 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 599/1189] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Sep 25 04:24:55 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:24:55.289905 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a11a7ef-471d-4903-b89b-c1f51f72a7cd None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:55.388556 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a11a7ef-471d-4903-b89b-c1f51f72a7cd None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:55.421769 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-673cd416-8bb8-4733-884d-a28495ad48bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:55.422391 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-673cd416-8bb8-4733-884d-a28495ad48bd None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:55.422673 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-673cd416-8bb8-4733-884d-a28495ad48bd None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:55.422948 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-673cd416-8bb8-4733-884d-a28495ad48bd None admin] PATH_INFO: `/v3/users/d43efa97e9bb47289259bf298126bef0` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:55.424056 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-673cd416-8bb8-4733-884d-a28495ad48bd None admin] RBAC: Authorizing `identity:get_user(user_id=d43efa97e9bb47289259bf298126bef0)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:55.425514 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-673cd416-8bb8-4733-884d-a28495ad48bd None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:55.427078 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 591/1190] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Sep 25 04:24:55 2022] GET /identity/v3/users/d43efa97e9bb47289259bf298126bef0 => generated 311 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:24:55.432795 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c6ff340-362e-4518-b895-e0c5c153d1c5 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:55.489627 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:24:55.539097 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c6ff340-362e-4518-b895-e0c5c153d1c5 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:55.570868 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca3d4d8e-7c4d-43c8-be07-50ca03982f01 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:55.571499 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca3d4d8e-7c4d-43c8-be07-50ca03982f01 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:55.571780 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca3d4d8e-7c4d-43c8-be07-50ca03982f01 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:55.572051 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca3d4d8e-7c4d-43c8-be07-50ca03982f01 None admin] PATH_INFO: `/v3/projects/da9e376f67414d7096edc245ae2e0b1c` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:55.573193 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca3d4d8e-7c4d-43c8-be07-50ca03982f01 None admin] RBAC: Authorizing `identity:get_project(project_id=da9e376f67414d7096edc245ae2e0b1c)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:55.574612 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca3d4d8e-7c4d-43c8-be07-50ca03982f01 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:55.577173 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 600/1191] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Sep 25 04:24:55 2022] GET /identity/v3/projects/da9e376f67414d7096edc245ae2e0b1c => generated 361 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:24:55.583160 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-673cd416-8bb8-4733-884d-a28495ad48bd None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:55.627437 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:24:55.680282 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-673cd416-8bb8-4733-884d-a28495ad48bd None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:55.713939 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea302d45-20e2-41e4-b47a-06269bdd2a8e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:55.714595 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea302d45-20e2-41e4-b47a-06269bdd2a8e None admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:55.714880 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea302d45-20e2-41e4-b47a-06269bdd2a8e None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:55.715152 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea302d45-20e2-41e4-b47a-06269bdd2a8e None admin] PATH_INFO: `/v3/projects/da9e376f67414d7096edc245ae2e0b1c/users/d43efa97e9bb47289259bf298126bef0/roles/c51fe7c5ad804040879acfb54269455f` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:55.717093 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea302d45-20e2-41e4-b47a-06269bdd2a8e None admin] RBAC: Authorizing `identity:create_grant(project_id=da9e376f67414d7096edc245ae2e0b1c, user_id=d43efa97e9bb47289259bf298126bef0, role_id=c51fe7c5ad804040879acfb54269455f)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:55.718510 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea302d45-20e2-41e4-b47a-06269bdd2a8e None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:55.730657 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 592/1192] 10.222.0.25 () {58 vars in 1439 bytes} [Sun Sep 25 04:24:55 2022] PUT /identity/v3/projects/da9e376f67414d7096edc245ae2e0b1c/users/d43efa97e9bb47289259bf298126bef0/roles/c51fe7c5ad804040879acfb54269455f => generated 0 bytes in 148 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:24:57.859575 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c89c4c25-a6b8-434b-bcb9-0c63bff341aa None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:57.860082 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c89c4c25-a6b8-434b-bcb9-0c63bff341aa None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:57.860082 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c89c4c25-a6b8-434b-bcb9-0c63bff341aa None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:57.861398 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 601/1193] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:24:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:24:57.866700 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d0428e4-3c07-400e-9ab1-03037f1d6be5 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:57.867112 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d0428e4-3c07-400e-9ab1-03037f1d6be5 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:57.867251 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d0428e4-3c07-400e-9ab1-03037f1d6be5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:57.885550 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-0d0428e4-3c07-400e-9ab1-03037f1d6be5 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:24:57.886728 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-0d0428e4-3c07-400e-9ab1-03037f1d6be5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:24:57.951184 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 593/1194] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:24:57 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 04:24:58.221765 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c89c4c25-a6b8-434b-bcb9-0c63bff341aa None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:58.320659 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c89c4c25-a6b8-434b-bcb9-0c63bff341aa None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:58.337085 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:24:58.355073 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd7fae8d-90c6-4226-bdb6-fe39e0f89bea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:58.355699 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd7fae8d-90c6-4226-bdb6-fe39e0f89bea None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:58.355980 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd7fae8d-90c6-4226-bdb6-fe39e0f89bea None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:58.356309 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd7fae8d-90c6-4226-bdb6-fe39e0f89bea None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:58.366695 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd7fae8d-90c6-4226-bdb6-fe39e0f89bea None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 04:24:58.371486 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-cd7fae8d-90c6-4226-bdb6-fe39e0f89bea None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 04:24:58.371486 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:24:58.371486 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:24:58.371486 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:24:58.371486 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:24:58.371486 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:24:58.371486 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:24:58.371486 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:24:58.371486 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:24:58.371486 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:24:58.371486 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:24:58.371486 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:24:58.371486 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 04:24:58.371486 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 04:24:58.371486 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 04:24:58.371486 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 04:24:58.372881 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 04:24:58.372881 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 04:24:58.372881 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 04:24:58.372881 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 04:24:58.372881 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:24:58.372881 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:24:58.372881 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:24:58.372881 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:24:58.372881 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:24:58.372881 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:24:58.372881 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:24:58.372881 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:24:58.372881 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:24:58.372881 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:24:58.372881 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:24:58.372881 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:24:58.373763 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:24:58.373763 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:24:58.373763 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:24:58.373763 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:24:58.373763 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 04:24:58.373763 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 04:24:58.373763 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 04:24:58.373763 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 04:24:58.373763 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 04:24:58.373763 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 04:24:58.373763 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 04:24:58.373763 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:24:58.373763 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 602/1195] 10.222.0.25 () {56 vars in 1096 bytes} [Sun Sep 25 04:24:58 2022] GET /identity/v3/roles/admin => generated 83 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:24:58.378582 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d0428e4-3c07-400e-9ab1-03037f1d6be5 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:58.411537 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:24:58.478616 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d0428e4-3c07-400e-9ab1-03037f1d6be5 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:58.511216 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6877ba13-c293-47ba-8410-73f812cdcf05 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:58.511815 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6877ba13-c293-47ba-8410-73f812cdcf05 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:58.512095 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6877ba13-c293-47ba-8410-73f812cdcf05 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:58.512368 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6877ba13-c293-47ba-8410-73f812cdcf05 None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:58.513658 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6877ba13-c293-47ba-8410-73f812cdcf05 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:24:58.513959 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6877ba13-c293-47ba-8410-73f812cdcf05 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:58.515401 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6877ba13-c293-47ba-8410-73f812cdcf05 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:58.523126 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 594/1196] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Sep 25 04:24:58 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:24:58.529634 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd7fae8d-90c6-4226-bdb6-fe39e0f89bea None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:58.625527 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd7fae8d-90c6-4226-bdb6-fe39e0f89bea None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:58.658335 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-439ec66e-660b-4800-af95-d1727eeaa842 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:58.658951 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-439ec66e-660b-4800-af95-d1727eeaa842 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:58.659229 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-439ec66e-660b-4800-af95-d1727eeaa842 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:58.659501 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-439ec66e-660b-4800-af95-d1727eeaa842 None admin] PATH_INFO: `/v3/users/d43efa97e9bb47289259bf298126bef0` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:58.660611 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-439ec66e-660b-4800-af95-d1727eeaa842 None admin] RBAC: Authorizing `identity:get_user(user_id=d43efa97e9bb47289259bf298126bef0)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:58.662053 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-439ec66e-660b-4800-af95-d1727eeaa842 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:58.663605 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 603/1197] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Sep 25 04:24:58 2022] GET /identity/v3/users/d43efa97e9bb47289259bf298126bef0 => generated 311 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:24:58.669728 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6877ba13-c293-47ba-8410-73f812cdcf05 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:58.761778 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:24:58.769587 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6877ba13-c293-47ba-8410-73f812cdcf05 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:58.802019 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39e27067-377d-4d84-8e88-3dcb92d737c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:58.802614 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39e27067-377d-4d84-8e88-3dcb92d737c3 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:58.802894 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39e27067-377d-4d84-8e88-3dcb92d737c3 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:58.803167 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39e27067-377d-4d84-8e88-3dcb92d737c3 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:58.868651 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39e27067-377d-4d84-8e88-3dcb92d737c3 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:58.870083 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39e27067-377d-4d84-8e88-3dcb92d737c3 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:58.959298 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:24:58.966301 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 595/1198] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Sep 25 04:24:58 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 297 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 25 04:24:58.972290 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-439ec66e-660b-4800-af95-d1727eeaa842 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:59.016910 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:24:59.069925 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-439ec66e-660b-4800-af95-d1727eeaa842 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:59.102760 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1319d0eb-2107-49a0-97a6-0585c54dd131 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:59.103381 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1319d0eb-2107-49a0-97a6-0585c54dd131 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:59.103660 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1319d0eb-2107-49a0-97a6-0585c54dd131 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:59.103932 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1319d0eb-2107-49a0-97a6-0585c54dd131 None admin] PATH_INFO: `/v3/projects/da9e376f67414d7096edc245ae2e0b1c` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:59.105062 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1319d0eb-2107-49a0-97a6-0585c54dd131 None admin] RBAC: Authorizing `identity:get_project(project_id=da9e376f67414d7096edc245ae2e0b1c)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:59.106663 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1319d0eb-2107-49a0-97a6-0585c54dd131 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:59.108272 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 604/1199] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Sep 25 04:24:58 2022] GET /identity/v3/projects/da9e376f67414d7096edc245ae2e0b1c => generated 361 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:24:59.114342 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39e27067-377d-4d84-8e88-3dcb92d737c3 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:24:59.209108 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39e27067-377d-4d84-8e88-3dcb92d737c3 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:24:59.241806 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7d9acc8-cce4-4085-8e27-e98f0690db23 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:24:59.242393 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7d9acc8-cce4-4085-8e27-e98f0690db23 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:24:59.242673 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7d9acc8-cce4-4085-8e27-e98f0690db23 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:24:59.242942 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7d9acc8-cce4-4085-8e27-e98f0690db23 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:24:59.243644 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7d9acc8-cce4-4085-8e27-e98f0690db23 None admin] RBAC: Adding query filter params (role.id=c51fe7c5ad804040879acfb54269455f, scope.project.id=da9e376f67414d7096edc245ae2e0b1c, user.id=d43efa97e9bb47289259bf298126bef0) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:24:59.243948 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7d9acc8-cce4-4085-8e27-e98f0690db23 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:24:59.245388 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7d9acc8-cce4-4085-8e27-e98f0690db23 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:24:59.252709 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 596/1200] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Sep 25 04:24:59 2022] GET /identity/v3/role_assignments?user.id=d43efa97e9bb47289259bf298126bef0&scope.project.id=da9e376f67414d7096edc245ae2e0b1c&role.id=c51fe7c5ad804040879acfb54269455f => generated 608 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:25:09.098688 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=44 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:25:09.250086 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=70 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:25:54.908959 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ac98a36-f3b5-407e-8720-15d6239acdf0 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:25:54.909561 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ac98a36-f3b5-407e-8720-15d6239acdf0 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:25:54.909561 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ac98a36-f3b5-407e-8720-15d6239acdf0 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:25:54.910894 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 605/1201] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:25:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:25:54.920674 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70686fde-38a0-4537-b596-5371f34d4a09 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:25:54.920674 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70686fde-38a0-4537-b596-5371f34d4a09 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:25:54.920674 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70686fde-38a0-4537-b596-5371f34d4a09 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:25:54.941860 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-70686fde-38a0-4537-b596-5371f34d4a09 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:25:54.943257 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-70686fde-38a0-4537-b596-5371f34d4a09 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:25:55.039128 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 597/1202] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:25:54 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 25 04:25:55.326822 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ac98a36-f3b5-407e-8720-15d6239acdf0 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:25:55.468559 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ac98a36-f3b5-407e-8720-15d6239acdf0 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:25:55.516411 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-698ad638-9557-441f-9511-da018872ea65 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:25:55.517187 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-698ad638-9557-441f-9511-da018872ea65 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:25:55.517506 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-698ad638-9557-441f-9511-da018872ea65 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:25:55.517869 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-698ad638-9557-441f-9511-da018872ea65 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:25:55.530146 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-698ad638-9557-441f-9511-da018872ea65 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:25:55.531779 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-698ad638-9557-441f-9511-da018872ea65 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:25:55.548495 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-698ad638-9557-441f-9511-da018872ea65 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:25:55.548495 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:25:55.548495 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:25:55.548495 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:25:55.548495 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:25:55.548495 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:25:55.548495 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:25:55.548495 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:25:55.548495 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:25:55.548495 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:25:55.548495 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:25:55.548495 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:25:55.548495 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 04:25:55.548495 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 04:25:55.548495 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 04:25:55.548495 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 04:25:55.549658 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:25:55.549658 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:25:55.549658 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:25:55.549658 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:25:55.549658 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:25:55.549658 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:25:55.549658 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:25:55.549658 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:25:55.549658 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:25:55.549658 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:25:55.549658 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:25:55.549658 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:25:55.549658 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:25:55.549658 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:25:55.549658 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:25:55.549658 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:25:55.549658 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 04:25:55.550721 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 04:25:55.550721 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:25:55.550721 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:25:55.550721 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 606/1203] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:25:55 2022] GET /identity/v3/domains/Default => generated 87 bytes in 224 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:25:55.556472 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70686fde-38a0-4537-b596-5371f34d4a09 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:25:55.694388 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70686fde-38a0-4537-b596-5371f34d4a09 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:25:55.735928 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:25:55.742276 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8c09995-453d-4a03-b537-bc78592a8176 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:25:55.742905 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8c09995-453d-4a03-b537-bc78592a8176 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:25:55.743222 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8c09995-453d-4a03-b537-bc78592a8176 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:25:55.743508 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8c09995-453d-4a03-b537-bc78592a8176 None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:25:55.744135 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8c09995-453d-4a03-b537-bc78592a8176 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:25:55.744492 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8c09995-453d-4a03-b537-bc78592a8176 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:25:55.747030 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8c09995-453d-4a03-b537-bc78592a8176 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:25:55.760693 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 598/1204] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Sep 25 04:25:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:25:55.766816 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-698ad638-9557-441f-9511-da018872ea65 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:25:55.828366 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:25:55.880744 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-698ad638-9557-441f-9511-da018872ea65 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:25:55.935116 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9804989-87e2-43f0-bf55-43a2dbcb447f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:25:55.935919 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9804989-87e2-43f0-bf55-43a2dbcb447f None admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:25:55.936204 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9804989-87e2-43f0-bf55-43a2dbcb447f None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:25:55.936506 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9804989-87e2-43f0-bf55-43a2dbcb447f None admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:25:55.937206 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9804989-87e2-43f0-bf55-43a2dbcb447f None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:25:55.938647 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9804989-87e2-43f0-bf55-43a2dbcb447f None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:25:55.976749 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 607/1205] 10.222.0.25 () {60 vars in 1132 bytes} [Sun Sep 25 04:25:55 2022] POST /identity/v3/users => generated 251 bytes in 199 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:25:58.535552 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aee6d82-3574-486c-b273-cdf73e995fce None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:25:58.535552 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aee6d82-3574-486c-b273-cdf73e995fce None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:25:58.535552 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aee6d82-3574-486c-b273-cdf73e995fce None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:25:58.536518 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 599/1206] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:25:58 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:25:58.546101 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1645e202-c7a9-4cac-87a3-dde38045204a None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:25:58.546613 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1645e202-c7a9-4cac-87a3-dde38045204a None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:25:58.546886 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1645e202-c7a9-4cac-87a3-dde38045204a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:25:58.565919 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-1645e202-c7a9-4cac-87a3-dde38045204a None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:25:58.567146 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-1645e202-c7a9-4cac-87a3-dde38045204a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:25:58.648219 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 608/1207] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:25:58 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 04:25:58.932568 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aee6d82-3574-486c-b273-cdf73e995fce None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:25:59.052925 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aee6d82-3574-486c-b273-cdf73e995fce None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:25:59.097528 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-306ca234-dc16-4dee-9aba-59e055936edc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:25:59.098221 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-306ca234-dc16-4dee-9aba-59e055936edc None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:25:59.098552 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-306ca234-dc16-4dee-9aba-59e055936edc None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:25:59.098862 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-306ca234-dc16-4dee-9aba-59e055936edc None admin] PATH_INFO: `/v3/roles/service` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:25:59.112100 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-306ca234-dc16-4dee-9aba-59e055936edc None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 04:25:59.118104 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-306ca234-dc16-4dee-9aba-59e055936edc None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 25 04:25:59.118104 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:25:59.118104 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:25:59.118104 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:25:59.118104 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:25:59.118104 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:25:59.118104 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:25:59.118104 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:25:59.118104 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:25:59.118104 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:25:59.118104 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:25:59.118104 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:25:59.118104 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 04:25:59.118104 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 04:25:59.118104 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 04:25:59.118104 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 04:25:59.120031 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 04:25:59.120031 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 04:25:59.120031 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 04:25:59.120031 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 04:25:59.120031 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:25:59.120031 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:25:59.120031 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:25:59.120031 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:25:59.120031 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:25:59.120031 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:25:59.120031 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:25:59.120031 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:25:59.120031 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:25:59.120031 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:25:59.120031 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:25:59.120031 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:25:59.121846 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:25:59.121846 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:25:59.121846 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:25:59.121846 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:25:59.121846 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 04:25:59.121846 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 04:25:59.121846 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 04:25:59.121846 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 04:25:59.121846 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 04:25:59.121846 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 04:25:59.121846 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 25 04:25:59.121846 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:25:59.121846 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 600/1208] 10.222.0.25 () {56 vars in 1102 bytes} [Sun Sep 25 04:25:58 2022] GET /identity/v3/roles/service => generated 85 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:25:59.127511 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1645e202-c7a9-4cac-87a3-dde38045204a None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:25:59.171948 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:25:59.235623 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1645e202-c7a9-4cac-87a3-dde38045204a None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:25:59.274907 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d1cf4a2-3e41-4d60-a969-98d9f4fb7702 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:25:59.275529 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d1cf4a2-3e41-4d60-a969-98d9f4fb7702 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:25:59.275810 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d1cf4a2-3e41-4d60-a969-98d9f4fb7702 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:25:59.276082 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d1cf4a2-3e41-4d60-a969-98d9f4fb7702 None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:25:59.277531 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d1cf4a2-3e41-4d60-a969-98d9f4fb7702 None admin] RBAC: Adding query filter params (name=service) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:25:59.277723 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d1cf4a2-3e41-4d60-a969-98d9f4fb7702 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:25:59.278981 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d1cf4a2-3e41-4d60-a969-98d9f4fb7702 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:25:59.287371 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 609/1209] 10.222.0.25 () {56 vars in 1103 bytes} [Sun Sep 25 04:25:59 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:25:59.293441 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-306ca234-dc16-4dee-9aba-59e055936edc None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:25:59.351172 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:25:59.411460 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-306ca234-dc16-4dee-9aba-59e055936edc None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:25:59.458994 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cffcbea-fd3c-4c9a-bd6c-5291b7974b12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:25:59.459973 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cffcbea-fd3c-4c9a-bd6c-5291b7974b12 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:25:59.460569 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cffcbea-fd3c-4c9a-bd6c-5291b7974b12 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:25:59.461030 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cffcbea-fd3c-4c9a-bd6c-5291b7974b12 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:25:59.474285 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cffcbea-fd3c-4c9a-bd6c-5291b7974b12 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:25:59.475961 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cffcbea-fd3c-4c9a-bd6c-5291b7974b12 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:25:59.494747 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-9cffcbea-fd3c-4c9a-bd6c-5291b7974b12 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:25:59.494747 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:25:59.494747 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:25:59.494747 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:25:59.494747 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:25:59.494747 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:25:59.494747 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:25:59.494747 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:25:59.494747 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:25:59.494747 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:25:59.494747 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:25:59.494747 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:25:59.494747 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 04:25:59.494747 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 04:25:59.494747 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 04:25:59.494747 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 04:25:59.496365 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:25:59.496365 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:25:59.496365 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:25:59.496365 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:25:59.496365 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:25:59.496365 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:25:59.496365 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:25:59.496365 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:25:59.496365 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:25:59.496365 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:25:59.496365 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:25:59.496365 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:25:59.496365 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:25:59.496365 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:25:59.496365 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:25:59.496365 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:25:59.496365 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 04:25:59.502164 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 04:25:59.502164 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:25:59.502164 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:25:59.502164 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 601/1210] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:25:59 2022] GET /identity/v3/domains/Default => generated 87 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:25:59.503957 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d1cf4a2-3e41-4d60-a969-98d9f4fb7702 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:25:59.615053 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d1cf4a2-3e41-4d60-a969-98d9f4fb7702 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:25:59.617921 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:25:59.654835 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7560a6d-d5d1-4837-a6f7-8648b57a9d77 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:25:59.655535 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7560a6d-d5d1-4837-a6f7-8648b57a9d77 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:25:59.655836 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7560a6d-d5d1-4837-a6f7-8648b57a9d77 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:25:59.656219 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7560a6d-d5d1-4837-a6f7-8648b57a9d77 None admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:25:59.656992 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7560a6d-d5d1-4837-a6f7-8648b57a9d77 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:25:59.657302 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7560a6d-d5d1-4837-a6f7-8648b57a9d77 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:25:59.658669 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7560a6d-d5d1-4837-a6f7-8648b57a9d77 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:25:59.672678 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 610/1211] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Sep 25 04:25:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:25:59.678537 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cffcbea-fd3c-4c9a-bd6c-5291b7974b12 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:25:59.790633 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cffcbea-fd3c-4c9a-bd6c-5291b7974b12 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:25:59.792121 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:25:59.834603 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cb56339-40e1-4468-b270-45bd9a097852 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:25:59.835268 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb56339-40e1-4468-b270-45bd9a097852 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:25:59.835547 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb56339-40e1-4468-b270-45bd9a097852 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:25:59.835824 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb56339-40e1-4468-b270-45bd9a097852 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:25:59.845389 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cb56339-40e1-4468-b270-45bd9a097852 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:25:59.846866 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cb56339-40e1-4468-b270-45bd9a097852 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:25:59.859352 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-1cb56339-40e1-4468-b270-45bd9a097852 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 25 04:25:59.859352 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:25:59.859352 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:25:59.859352 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:25:59.859352 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:25:59.859352 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:25:59.859352 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:25:59.859352 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:25:59.859352 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:25:59.859352 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:25:59.859352 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:25:59.859352 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:25:59.859352 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 25 04:25:59.859352 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 25 04:25:59.859352 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 25 04:25:59.859352 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 25 04:25:59.861100 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:25:59.861100 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:25:59.861100 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 25 04:25:59.861100 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 04:25:59.861100 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 25 04:25:59.861100 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 04:25:59.861100 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:25:59.861100 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:25:59.861100 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:25:59.861100 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:25:59.861100 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:25:59.861100 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:25:59.861100 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:25:59.861100 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:25:59.861100 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:25:59.861100 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:25:59.861100 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:25:59.861100 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:25:59.862443 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:25:59.862443 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:25:59.862443 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 25 04:25:59.862443 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 25 04:25:59.862443 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 25 04:25:59.862443 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 25 04:25:59.862443 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 25 04:25:59.862443 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 25 04:25:59.862443 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 25 04:25:59.862443 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:25:59.866218 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 602/1212] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:25:59 2022] GET /identity/v3/users/placement => generated 87 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:25:59.867817 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7560a6d-d5d1-4837-a6f7-8648b57a9d77 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:25:59.981767 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7560a6d-d5d1-4837-a6f7-8648b57a9d77 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:26:00.018885 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e8a73d7-ca3d-42a4-aa0e-0ffad00363b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:26:00.019522 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e8a73d7-ca3d-42a4-aa0e-0ffad00363b2 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:26:00.019803 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e8a73d7-ca3d-42a4-aa0e-0ffad00363b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:26:00.020077 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e8a73d7-ca3d-42a4-aa0e-0ffad00363b2 None admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:26:00.020840 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e8a73d7-ca3d-42a4-aa0e-0ffad00363b2 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:26:00.021150 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e8a73d7-ca3d-42a4-aa0e-0ffad00363b2 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:26:00.022573 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e8a73d7-ca3d-42a4-aa0e-0ffad00363b2 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:26:00.041790 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 611/1213] 10.222.0.25 () {56 vars in 1143 bytes} [Sun Sep 25 04:25:59 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:26:00.048108 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cb56339-40e1-4468-b270-45bd9a097852 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:26:00.178067 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cb56339-40e1-4468-b270-45bd9a097852 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:26:00.219178 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f97d8c4-3261-4b57-b791-cc2476d27268 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:26:00.219938 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f97d8c4-3261-4b57-b791-cc2476d27268 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:26:00.220322 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f97d8c4-3261-4b57-b791-cc2476d27268 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:26:00.220753 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f97d8c4-3261-4b57-b791-cc2476d27268 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:26:00.256080 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:26:00.300333 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f97d8c4-3261-4b57-b791-cc2476d27268 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:26:00.301840 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f97d8c4-3261-4b57-b791-cc2476d27268 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:26:00.409620 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 603/1214] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Sep 25 04:26:00 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 362 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 25 04:26:00.415300 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e8a73d7-ca3d-42a4-aa0e-0ffad00363b2 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:26:00.436471 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:26:00.534703 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e8a73d7-ca3d-42a4-aa0e-0ffad00363b2 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:26:00.576495 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d261e5b-0b28-4335-8e5d-6640ffa66d1f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:26:00.577121 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d261e5b-0b28-4335-8e5d-6640ffa66d1f None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:26:00.577409 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d261e5b-0b28-4335-8e5d-6640ffa66d1f None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:26:00.577682 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d261e5b-0b28-4335-8e5d-6640ffa66d1f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:26:00.590069 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d261e5b-0b28-4335-8e5d-6640ffa66d1f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:26:00.591690 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d261e5b-0b28-4335-8e5d-6640ffa66d1f None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:26:00.612360 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-4d261e5b-0b28-4335-8e5d-6640ffa66d1f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:26:00.612360 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:26:00.612360 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:26:00.612360 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:26:00.612360 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:26:00.612360 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:26:00.612360 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:26:00.612360 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:26:00.612360 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:26:00.612360 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:26:00.612360 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:26:00.612360 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:26:00.612360 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 04:26:00.612360 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 04:26:00.612360 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 04:26:00.612360 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 04:26:00.613666 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:26:00.613666 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:26:00.613666 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:26:00.613666 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:26:00.613666 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:26:00.613666 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:26:00.613666 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:26:00.613666 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:26:00.613666 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:26:00.613666 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:26:00.613666 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:26:00.613666 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:26:00.613666 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:26:00.613666 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:26:00.613666 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:26:00.613666 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:26:00.613666 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 04:26:00.619812 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 04:26:00.619812 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:26:00.619812 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:26:00.619812 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 612/1215] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:26:00 2022] GET /identity/v3/domains/Default => generated 87 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:26:00.620100 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f97d8c4-3261-4b57-b791-cc2476d27268 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:26:00.694434 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:26:00.743051 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f97d8c4-3261-4b57-b791-cc2476d27268 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:26:00.783272 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-913b0cb2-3e49-4929-ac83-bb52744f2a18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:26:00.783906 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-913b0cb2-3e49-4929-ac83-bb52744f2a18 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:26:00.784313 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-913b0cb2-3e49-4929-ac83-bb52744f2a18 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:26:00.784499 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-913b0cb2-3e49-4929-ac83-bb52744f2a18 None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:26:00.785230 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-913b0cb2-3e49-4929-ac83-bb52744f2a18 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:26:00.785581 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-913b0cb2-3e49-4929-ac83-bb52744f2a18 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:26:00.787077 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-913b0cb2-3e49-4929-ac83-bb52744f2a18 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:26:00.807756 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 604/1216] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Sep 25 04:26:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:26:00.807949 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d261e5b-0b28-4335-8e5d-6640ffa66d1f None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:26:00.890923 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:26:00.932037 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d261e5b-0b28-4335-8e5d-6640ffa66d1f None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:26:00.975438 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-937b0292-b4cf-40d1-825f-cd93b36a856b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:26:00.976102 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-937b0292-b4cf-40d1-825f-cd93b36a856b None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:26:00.976422 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-937b0292-b4cf-40d1-825f-cd93b36a856b None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:26:00.976732 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-937b0292-b4cf-40d1-825f-cd93b36a856b None admin] PATH_INFO: `/v3/projects/service` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:26:00.985257 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-937b0292-b4cf-40d1-825f-cd93b36a856b None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:26:00.987017 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-937b0292-b4cf-40d1-825f-cd93b36a856b None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:26:01.003171 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-937b0292-b4cf-40d1-825f-cd93b36a856b None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 04:26:01.003171 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:26:01.003171 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:26:01.003171 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:26:01.003171 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:26:01.003171 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:26:01.003171 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:26:01.003171 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:26:01.003171 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:26:01.003171 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:26:01.003171 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:26:01.003171 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:26:01.003171 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 25 04:26:01.003171 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 25 04:26:01.003171 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 25 04:26:01.004370 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 25 04:26:01.004370 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:26:01.004370 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:26:01.004370 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:26:01.004370 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:26:01.004370 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:26:01.004370 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:26:01.004370 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:26:01.004370 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:26:01.004370 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:26:01.004370 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:26:01.004370 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:26:01.004370 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:26:01.004370 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:26:01.004370 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:26:01.004370 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:26:01.004370 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:26:01.005525 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 25 04:26:01.005525 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 25 04:26:01.005525 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 25 04:26:01.005525 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 25 04:26:01.005525 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 25 04:26:01.005525 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 25 04:26:01.005525 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 04:26:01.005525 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:26:01.005525 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 613/1217] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Sep 25 04:26:00 2022] GET /identity/v3/projects/service => generated 88 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:26:01.010940 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-913b0cb2-3e49-4929-ac83-bb52744f2a18 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:26:01.130695 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-913b0cb2-3e49-4929-ac83-bb52744f2a18 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:26:01.155485 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:26:01.169710 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9069b52-ba58-4b86-9cf0-c3f59236944a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:26:01.170397 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9069b52-ba58-4b86-9cf0-c3f59236944a None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:26:01.170686 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9069b52-ba58-4b86-9cf0-c3f59236944a None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:26:01.171001 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9069b52-ba58-4b86-9cf0-c3f59236944a None admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:26:01.171768 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9069b52-ba58-4b86-9cf0-c3f59236944a None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:26:01.172088 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9069b52-ba58-4b86-9cf0-c3f59236944a None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:26:01.173624 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9069b52-ba58-4b86-9cf0-c3f59236944a None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:26:01.188456 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 605/1218] 10.222.0.25 () {56 vars in 1148 bytes} [Sun Sep 25 04:26:01 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:26:01.195013 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-937b0292-b4cf-40d1-825f-cd93b36a856b None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:26:01.304705 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-937b0292-b4cf-40d1-825f-cd93b36a856b None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:26:01.337442 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:26:01.344938 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d18b627e-0412-4299-874b-6d24e39ab114 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:26:01.345613 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d18b627e-0412-4299-874b-6d24e39ab114 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:26:01.345891 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d18b627e-0412-4299-874b-6d24e39ab114 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:26:01.346246 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d18b627e-0412-4299-874b-6d24e39ab114 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:26:01.347108 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d18b627e-0412-4299-874b-6d24e39ab114 None admin] RBAC: Adding query filter params (role.id=9b44520f482c4423bd10568d0f86718c, scope.project.id=056ca1b1a9974a7581550de0d9217823, user.id=676d04ef5a984edcaad63d9156568660) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:26:01.347443 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d18b627e-0412-4299-874b-6d24e39ab114 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:26:01.348943 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d18b627e-0412-4299-874b-6d24e39ab114 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:26:01.355848 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 614/1219] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Sep 25 04:26:01 2022] GET /identity/v3/role_assignments?user.id=676d04ef5a984edcaad63d9156568660&scope.project.id=056ca1b1a9974a7581550de0d9217823&role.id=9b44520f482c4423bd10568d0f86718c => generated 259 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:26:03.613852 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2be6904-c7be-47f5-9ae7-d97d80a36463 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:26:03.614408 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2be6904-c7be-47f5-9ae7-d97d80a36463 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:26:03.614503 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2be6904-c7be-47f5-9ae7-d97d80a36463 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:26:03.615923 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 606/1220] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:26:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:26:03.622068 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddc3ee30-71a9-4b9e-a7e4-99b4386696f1 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:26:03.622546 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddc3ee30-71a9-4b9e-a7e4-99b4386696f1 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:26:03.622841 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddc3ee30-71a9-4b9e-a7e4-99b4386696f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:26:03.643521 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-ddc3ee30-71a9-4b9e-a7e4-99b4386696f1 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:26:03.644747 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-ddc3ee30-71a9-4b9e-a7e4-99b4386696f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:26:03.722188 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 615/1221] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:26:03 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 04:26:03.994985 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2be6904-c7be-47f5-9ae7-d97d80a36463 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:26:04.105979 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2be6904-c7be-47f5-9ae7-d97d80a36463 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:26:04.142029 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79b633e6-5601-447b-910c-2f75df02544f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:26:04.142705 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b633e6-5601-447b-910c-2f75df02544f None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:26:04.143189 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b633e6-5601-447b-910c-2f75df02544f None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:26:04.143378 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b633e6-5601-447b-910c-2f75df02544f None admin] PATH_INFO: `/v3/roles/service` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:26:04.155289 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79b633e6-5601-447b-910c-2f75df02544f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 04:26:04.160107 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-79b633e6-5601-447b-910c-2f75df02544f None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 25 04:26:04.160107 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:26:04.160107 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:26:04.160107 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:26:04.160107 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:26:04.160107 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:26:04.160107 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:26:04.160107 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:26:04.160107 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:26:04.160107 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:26:04.160107 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:26:04.160107 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:26:04.160107 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 04:26:04.160107 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 04:26:04.160107 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 04:26:04.160107 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 04:26:04.161298 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 04:26:04.161298 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 04:26:04.161298 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 04:26:04.161298 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 04:26:04.161298 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:26:04.161298 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:26:04.161298 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:26:04.161298 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:26:04.161298 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:26:04.161298 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:26:04.161298 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:26:04.161298 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:26:04.161298 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:26:04.161298 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:26:04.161298 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:26:04.161298 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:26:04.162203 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:26:04.162203 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:26:04.162203 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:26:04.162203 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:26:04.162203 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 04:26:04.162203 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 04:26:04.162203 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 04:26:04.162203 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 04:26:04.162203 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 04:26:04.162203 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 04:26:04.162203 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 25 04:26:04.162203 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:26:04.162203 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 607/1222] 10.222.0.25 () {56 vars in 1102 bytes} [Sun Sep 25 04:26:03 2022] GET /identity/v3/roles/service => generated 85 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:26:04.167351 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddc3ee30-71a9-4b9e-a7e4-99b4386696f1 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:26:04.271419 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddc3ee30-71a9-4b9e-a7e4-99b4386696f1 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:26:04.295634 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:26:04.306838 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31761d57-5b49-4046-963e-fa7475d12704 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:26:04.307455 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31761d57-5b49-4046-963e-fa7475d12704 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:26:04.307735 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31761d57-5b49-4046-963e-fa7475d12704 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:26:04.308048 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31761d57-5b49-4046-963e-fa7475d12704 None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:26:04.308912 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31761d57-5b49-4046-963e-fa7475d12704 None admin] RBAC: Adding query filter params (name=service) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:26:04.309218 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31761d57-5b49-4046-963e-fa7475d12704 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:26:04.310592 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31761d57-5b49-4046-963e-fa7475d12704 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:26:04.319056 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 616/1223] 10.222.0.25 () {56 vars in 1103 bytes} [Sun Sep 25 04:26:04 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:26:04.325553 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79b633e6-5601-447b-910c-2f75df02544f None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:26:04.370366 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:26:04.443001 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79b633e6-5601-447b-910c-2f75df02544f None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:26:04.485903 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87534cbd-1d94-41ea-a69b-e963ddac0c20 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:26:04.486699 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87534cbd-1d94-41ea-a69b-e963ddac0c20 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:26:04.487053 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87534cbd-1d94-41ea-a69b-e963ddac0c20 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:26:04.487482 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87534cbd-1d94-41ea-a69b-e963ddac0c20 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:26:04.499751 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87534cbd-1d94-41ea-a69b-e963ddac0c20 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:26:04.501256 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87534cbd-1d94-41ea-a69b-e963ddac0c20 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:26:04.517519 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-87534cbd-1d94-41ea-a69b-e963ddac0c20 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:26:04.517519 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:26:04.517519 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:26:04.517519 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:26:04.517519 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:26:04.517519 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:26:04.517519 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:26:04.517519 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:26:04.517519 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:26:04.517519 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:26:04.517519 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:26:04.517519 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:26:04.517519 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 04:26:04.517519 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 04:26:04.517519 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 04:26:04.517519 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 04:26:04.518762 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:26:04.518762 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:26:04.518762 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:26:04.518762 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:26:04.518762 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:26:04.518762 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:26:04.518762 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:26:04.518762 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:26:04.518762 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:26:04.518762 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:26:04.518762 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:26:04.518762 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:26:04.518762 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:26:04.518762 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:26:04.518762 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:26:04.518762 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:26:04.518762 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 04:26:04.519681 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 04:26:04.519681 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:26:04.519681 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:26:04.519681 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 608/1224] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:26:04 2022] GET /identity/v3/domains/Default => generated 87 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:26:04.524996 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31761d57-5b49-4046-963e-fa7475d12704 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:26:04.641140 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31761d57-5b49-4046-963e-fa7475d12704 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:26:04.676217 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00db7df6-b5c2-4c8b-a85e-dd3c48753166 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:26:04.676881 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00db7df6-b5c2-4c8b-a85e-dd3c48753166 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:26:04.677149 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00db7df6-b5c2-4c8b-a85e-dd3c48753166 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:26:04.677421 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00db7df6-b5c2-4c8b-a85e-dd3c48753166 None admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:26:04.678063 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00db7df6-b5c2-4c8b-a85e-dd3c48753166 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:26:04.678416 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00db7df6-b5c2-4c8b-a85e-dd3c48753166 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:26:04.679731 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00db7df6-b5c2-4c8b-a85e-dd3c48753166 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:26:04.694527 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 617/1225] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Sep 25 04:26:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:26:04.701129 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87534cbd-1d94-41ea-a69b-e963ddac0c20 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:26:04.799943 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:26:04.807539 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87534cbd-1d94-41ea-a69b-e963ddac0c20 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:26:04.844189 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-962cf4db-99f4-45c6-a261-8318c46ed24d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:26:04.844856 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-962cf4db-99f4-45c6-a261-8318c46ed24d None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:26:04.845135 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-962cf4db-99f4-45c6-a261-8318c46ed24d None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:26:04.845400 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-962cf4db-99f4-45c6-a261-8318c46ed24d None admin] PATH_INFO: `/v3/users/placement` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:26:04.853662 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-962cf4db-99f4-45c6-a261-8318c46ed24d None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:26:04.855154 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-962cf4db-99f4-45c6-a261-8318c46ed24d None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:26:04.865977 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-962cf4db-99f4-45c6-a261-8318c46ed24d None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 25 04:26:04.865977 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:26:04.865977 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:26:04.865977 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:26:04.865977 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:26:04.865977 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:26:04.865977 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:26:04.865977 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:26:04.865977 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:26:04.865977 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:26:04.865977 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:26:04.865977 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:26:04.865977 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 25 04:26:04.865977 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 25 04:26:04.865977 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 25 04:26:04.865977 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 25 04:26:04.867141 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:26:04.867141 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:26:04.867141 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 25 04:26:04.867141 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 04:26:04.867141 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 25 04:26:04.867141 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 04:26:04.867141 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:26:04.867141 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:26:04.867141 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:26:04.867141 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:26:04.867141 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:26:04.867141 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:26:04.867141 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:26:04.867141 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:26:04.867141 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:26:04.867141 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:26:04.867141 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:26:04.867141 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:26:04.868499 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:26:04.868499 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:26:04.868499 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 25 04:26:04.868499 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 25 04:26:04.868499 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 25 04:26:04.868499 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 25 04:26:04.868499 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 25 04:26:04.868499 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 25 04:26:04.868499 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 25 04:26:04.868499 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:26:04.872488 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 609/1226] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:26:04 2022] GET /identity/v3/users/placement => generated 87 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:26:04.873472 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00db7df6-b5c2-4c8b-a85e-dd3c48753166 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:26:04.924986 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:26:04.986256 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00db7df6-b5c2-4c8b-a85e-dd3c48753166 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:26:05.027421 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9745d775-3243-4253-bc07-41427d7ae5c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:26:05.028022 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9745d775-3243-4253-bc07-41427d7ae5c7 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:26:05.028305 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9745d775-3243-4253-bc07-41427d7ae5c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:26:05.028575 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9745d775-3243-4253-bc07-41427d7ae5c7 None admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:26:05.029337 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9745d775-3243-4253-bc07-41427d7ae5c7 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:26:05.029642 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9745d775-3243-4253-bc07-41427d7ae5c7 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:26:05.031064 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9745d775-3243-4253-bc07-41427d7ae5c7 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:26:05.053158 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 618/1227] 10.222.0.25 () {56 vars in 1143 bytes} [Sun Sep 25 04:26:04 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:26:05.059003 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-962cf4db-99f4-45c6-a261-8318c46ed24d None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:26:05.173600 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-962cf4db-99f4-45c6-a261-8318c46ed24d None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:26:05.212038 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d6b402f-c712-48be-8dd3-0f6bf6dd58b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:26:05.212701 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d6b402f-c712-48be-8dd3-0f6bf6dd58b5 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:26:05.212975 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d6b402f-c712-48be-8dd3-0f6bf6dd58b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:26:05.213272 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d6b402f-c712-48be-8dd3-0f6bf6dd58b5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:26:05.227898 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d6b402f-c712-48be-8dd3-0f6bf6dd58b5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:26:05.229515 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d6b402f-c712-48be-8dd3-0f6bf6dd58b5 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:26:05.246825 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-9d6b402f-c712-48be-8dd3-0f6bf6dd58b5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:26:05.246825 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:26:05.246825 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:26:05.246825 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:26:05.246825 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:26:05.246825 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:26:05.246825 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:26:05.246825 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:26:05.246825 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:26:05.246825 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:26:05.246825 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:26:05.246825 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:26:05.246825 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 04:26:05.246825 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 04:26:05.246825 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 04:26:05.246825 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 04:26:05.252697 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:26:05.252697 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:26:05.252697 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:26:05.252697 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:26:05.252697 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:26:05.252697 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:26:05.252697 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:26:05.252697 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:26:05.252697 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:26:05.252697 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:26:05.252697 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:26:05.252697 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:26:05.252697 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:26:05.252697 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:26:05.252697 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:26:05.252697 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:26:05.252697 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 04:26:05.253807 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 04:26:05.253807 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:26:05.253807 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:26:05.253807 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 610/1228] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:26:05 2022] GET /identity/v3/domains/Default => generated 87 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:26:05.254052 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9745d775-3243-4253-bc07-41427d7ae5c7 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:26:05.360985 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9745d775-3243-4253-bc07-41427d7ae5c7 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:26:05.363473 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:26:05.398199 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05961167-68dd-4994-9083-e442de9d96c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:26:05.398868 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05961167-68dd-4994-9083-e442de9d96c4 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:26:05.399203 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05961167-68dd-4994-9083-e442de9d96c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:26:05.399519 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05961167-68dd-4994-9083-e442de9d96c4 None admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:26:05.400258 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05961167-68dd-4994-9083-e442de9d96c4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:26:05.400673 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05961167-68dd-4994-9083-e442de9d96c4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:26:05.402396 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05961167-68dd-4994-9083-e442de9d96c4 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:26:05.415649 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 619/1229] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Sep 25 04:26:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:26:05.422797 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d6b402f-c712-48be-8dd3-0f6bf6dd58b5 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:26:05.432011 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:26:05.532023 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d6b402f-c712-48be-8dd3-0f6bf6dd58b5 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:26:05.567339 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20c5be20-3174-412a-94ea-03a53ca5b95d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:26:05.567999 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c5be20-3174-412a-94ea-03a53ca5b95d None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:26:05.568293 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c5be20-3174-412a-94ea-03a53ca5b95d None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:26:05.568579 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c5be20-3174-412a-94ea-03a53ca5b95d None admin] PATH_INFO: `/v3/projects/service` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:26:05.575906 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20c5be20-3174-412a-94ea-03a53ca5b95d None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:26:05.577393 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20c5be20-3174-412a-94ea-03a53ca5b95d None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:26:05.588650 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-20c5be20-3174-412a-94ea-03a53ca5b95d None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 04:26:05.588650 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:26:05.588650 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:26:05.588650 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:26:05.588650 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:26:05.588650 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:26:05.588650 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:26:05.588650 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:26:05.588650 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:26:05.588650 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:26:05.588650 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:26:05.588650 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:26:05.588650 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 25 04:26:05.588650 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 25 04:26:05.588650 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 25 04:26:05.597544 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 25 04:26:05.597544 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:26:05.597544 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:26:05.597544 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:26:05.597544 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:26:05.597544 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:26:05.597544 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:26:05.597544 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:26:05.597544 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:26:05.597544 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:26:05.597544 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:26:05.597544 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:26:05.597544 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:26:05.597544 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:26:05.597544 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:26:05.597544 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:26:05.597544 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:26:05.598677 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 25 04:26:05.598677 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 25 04:26:05.598677 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 25 04:26:05.598677 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 25 04:26:05.598677 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 25 04:26:05.598677 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 25 04:26:05.598677 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 04:26:05.598677 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:26:05.598677 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 611/1230] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Sep 25 04:26:05 2022] GET /identity/v3/projects/service => generated 88 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 04:26:05.599281 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05961167-68dd-4994-9083-e442de9d96c4 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:26:05.711372 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05961167-68dd-4994-9083-e442de9d96c4 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:26:05.748877 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aee9e2b3-9dea-4d55-a1b7-0a4f09c400c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:26:05.749790 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aee9e2b3-9dea-4d55-a1b7-0a4f09c400c8 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:26:05.750171 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aee9e2b3-9dea-4d55-a1b7-0a4f09c400c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:26:05.750568 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aee9e2b3-9dea-4d55-a1b7-0a4f09c400c8 None admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:26:05.751650 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aee9e2b3-9dea-4d55-a1b7-0a4f09c400c8 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:26:05.751973 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aee9e2b3-9dea-4d55-a1b7-0a4f09c400c8 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:26:05.753421 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aee9e2b3-9dea-4d55-a1b7-0a4f09c400c8 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:26:05.766017 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 620/1231] 10.222.0.25 () {56 vars in 1148 bytes} [Sun Sep 25 04:26:05 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:26:05.771997 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20c5be20-3174-412a-94ea-03a53ca5b95d None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:26:05.841156 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:26:05.876866 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20c5be20-3174-412a-94ea-03a53ca5b95d None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:26:05.914611 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b15b643f-de7e-4b96-9bb1-da907ecdb6d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:26:05.915287 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b15b643f-de7e-4b96-9bb1-da907ecdb6d8 None admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:26:05.915571 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b15b643f-de7e-4b96-9bb1-da907ecdb6d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:26:05.915847 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b15b643f-de7e-4b96-9bb1-da907ecdb6d8 None admin] PATH_INFO: `/v3/projects/056ca1b1a9974a7581550de0d9217823/users/676d04ef5a984edcaad63d9156568660/roles/9b44520f482c4423bd10568d0f86718c` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:26:05.930929 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b15b643f-de7e-4b96-9bb1-da907ecdb6d8 None admin] RBAC: Authorizing `identity:create_grant(project_id=056ca1b1a9974a7581550de0d9217823, user_id=676d04ef5a984edcaad63d9156568660, role_id=9b44520f482c4423bd10568d0f86718c)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:26:05.932363 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b15b643f-de7e-4b96-9bb1-da907ecdb6d8 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:26:05.946221 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 612/1232] 10.222.0.25 () {58 vars in 1439 bytes} [Sun Sep 25 04:26:05 2022] PUT /identity/v3/projects/056ca1b1a9974a7581550de0d9217823/users/676d04ef5a984edcaad63d9156568660/roles/9b44520f482c4423bd10568d0f86718c => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:26:08.266751 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce1e098b-a17a-4d50-83b2-6709934d8151 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:26:08.267364 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce1e098b-a17a-4d50-83b2-6709934d8151 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:26:08.267364 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce1e098b-a17a-4d50-83b2-6709934d8151 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:26:08.269036 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 621/1233] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:26:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:26:08.275240 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-721327fb-cccd-4044-91d4-295cad179c27 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:26:08.275696 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-721327fb-cccd-4044-91d4-295cad179c27 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:26:08.275801 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-721327fb-cccd-4044-91d4-295cad179c27 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:26:08.296281 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-721327fb-cccd-4044-91d4-295cad179c27 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:26:08.297774 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-721327fb-cccd-4044-91d4-295cad179c27 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:26:08.373601 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 613/1234] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:26:08 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 04:26:08.672118 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce1e098b-a17a-4d50-83b2-6709934d8151 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:26:08.698277 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:26:08.782295 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce1e098b-a17a-4d50-83b2-6709934d8151 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:26:08.823051 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-187b77fe-0ae7-47ca-be5a-32e3896ebee5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:26:08.823849 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-187b77fe-0ae7-47ca-be5a-32e3896ebee5 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:26:08.824128 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-187b77fe-0ae7-47ca-be5a-32e3896ebee5 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:26:08.824396 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-187b77fe-0ae7-47ca-be5a-32e3896ebee5 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:26:08.836844 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-187b77fe-0ae7-47ca-be5a-32e3896ebee5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 04:26:08.843015 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-187b77fe-0ae7-47ca-be5a-32e3896ebee5 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 25 04:26:08.843015 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:26:08.843015 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:26:08.843015 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:26:08.843015 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:26:08.843015 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:26:08.843015 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:26:08.843015 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:26:08.843015 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:26:08.843015 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:26:08.843015 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:26:08.843015 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:26:08.843015 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 04:26:08.843015 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 04:26:08.843015 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 04:26:08.843015 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 04:26:08.844466 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 04:26:08.844466 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 04:26:08.844466 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 04:26:08.844466 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 04:26:08.844466 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:26:08.844466 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:26:08.844466 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:26:08.844466 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:26:08.844466 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:26:08.844466 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:26:08.844466 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:26:08.844466 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:26:08.844466 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:26:08.844466 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:26:08.844466 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:26:08.844466 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:26:08.845484 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:26:08.845484 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:26:08.845484 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:26:08.845484 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:26:08.845484 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 04:26:08.845484 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 04:26:08.845484 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 04:26:08.845484 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 04:26:08.845484 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 04:26:08.845484 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 04:26:08.845484 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 25 04:26:08.845484 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:26:08.845484 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 622/1235] 10.222.0.25 () {56 vars in 1102 bytes} [Sun Sep 25 04:26:08 2022] GET /identity/v3/roles/service => generated 85 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:26:08.851387 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-721327fb-cccd-4044-91d4-295cad179c27 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:26:08.899304 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:26:08.958538 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-721327fb-cccd-4044-91d4-295cad179c27 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:26:08.994568 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d30c4164-f011-4600-851d-e13a1ab827dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:26:08.995214 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d30c4164-f011-4600-851d-e13a1ab827dc None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:26:08.996013 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d30c4164-f011-4600-851d-e13a1ab827dc None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:26:08.998051 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d30c4164-f011-4600-851d-e13a1ab827dc None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:26:08.998744 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d30c4164-f011-4600-851d-e13a1ab827dc None admin] RBAC: Adding query filter params (name=service) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:26:08.999054 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d30c4164-f011-4600-851d-e13a1ab827dc None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:26:09.000438 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d30c4164-f011-4600-851d-e13a1ab827dc None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:26:09.008699 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 614/1236] 10.222.0.25 () {56 vars in 1103 bytes} [Sun Sep 25 04:26:08 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:26:09.015110 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-187b77fe-0ae7-47ca-be5a-32e3896ebee5 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:26:09.108110 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:26:09.130351 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-187b77fe-0ae7-47ca-be5a-32e3896ebee5 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:26:09.174908 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-498c130d-803a-4e94-bdcf-818ffd1b51a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:26:09.175551 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-498c130d-803a-4e94-bdcf-818ffd1b51a0 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:26:09.175829 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-498c130d-803a-4e94-bdcf-818ffd1b51a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:26:09.176111 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-498c130d-803a-4e94-bdcf-818ffd1b51a0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:26:09.189547 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-498c130d-803a-4e94-bdcf-818ffd1b51a0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:26:09.191242 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-498c130d-803a-4e94-bdcf-818ffd1b51a0 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:26:09.207202 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-498c130d-803a-4e94-bdcf-818ffd1b51a0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:26:09.207202 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:26:09.207202 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:26:09.207202 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:26:09.207202 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:26:09.207202 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:26:09.207202 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:26:09.207202 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:26:09.207202 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:26:09.207202 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:26:09.207202 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:26:09.207202 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:26:09.207202 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 04:26:09.207202 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 04:26:09.207202 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 04:26:09.207202 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 04:26:09.208744 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:26:09.208744 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:26:09.208744 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:26:09.208744 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:26:09.208744 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:26:09.208744 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:26:09.208744 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:26:09.208744 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:26:09.208744 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:26:09.208744 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:26:09.208744 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:26:09.208744 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:26:09.208744 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:26:09.208744 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:26:09.208744 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:26:09.208744 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:26:09.208744 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 04:26:09.210186 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 04:26:09.210186 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:26:09.210186 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:26:09.210186 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 623/1237] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:26:09 2022] GET /identity/v3/domains/Default => generated 87 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:26:09.214955 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d30c4164-f011-4600-851d-e13a1ab827dc None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:26:09.331832 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d30c4164-f011-4600-851d-e13a1ab827dc None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:26:09.342952 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:26:09.372660 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35d0393e-281e-4467-ba64-d108c843211f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:26:09.373387 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d0393e-281e-4467-ba64-d108c843211f None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:26:09.373693 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d0393e-281e-4467-ba64-d108c843211f None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:26:09.374029 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d0393e-281e-4467-ba64-d108c843211f None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:26:09.374706 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35d0393e-281e-4467-ba64-d108c843211f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:26:09.375017 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35d0393e-281e-4467-ba64-d108c843211f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:26:09.376557 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35d0393e-281e-4467-ba64-d108c843211f None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:26:09.390012 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 615/1238] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Sep 25 04:26:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:26:09.396056 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-498c130d-803a-4e94-bdcf-818ffd1b51a0 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:26:09.508523 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-498c130d-803a-4e94-bdcf-818ffd1b51a0 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:26:09.546255 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2573c08e-d94a-4d2b-8029-0afd929700cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:26:09.546999 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2573c08e-d94a-4d2b-8029-0afd929700cc None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:26:09.547282 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2573c08e-d94a-4d2b-8029-0afd929700cc None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:26:09.547553 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2573c08e-d94a-4d2b-8029-0afd929700cc None admin] PATH_INFO: `/v3/users/placement` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:26:09.553328 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:26:09.559253 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2573c08e-d94a-4d2b-8029-0afd929700cc None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:26:09.560911 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2573c08e-d94a-4d2b-8029-0afd929700cc None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:26:09.574701 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-2573c08e-d94a-4d2b-8029-0afd929700cc None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 25 04:26:09.574701 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:26:09.574701 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:26:09.574701 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:26:09.574701 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:26:09.574701 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:26:09.574701 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:26:09.574701 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:26:09.574701 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:26:09.574701 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:26:09.574701 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:26:09.574701 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:26:09.574701 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 25 04:26:09.574701 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 25 04:26:09.574701 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 25 04:26:09.574701 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 25 04:26:09.576063 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:26:09.576063 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:26:09.576063 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 25 04:26:09.576063 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 04:26:09.576063 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 25 04:26:09.576063 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 04:26:09.576063 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:26:09.576063 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:26:09.576063 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:26:09.576063 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:26:09.576063 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:26:09.576063 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:26:09.576063 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:26:09.576063 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:26:09.576063 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:26:09.576063 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:26:09.576063 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:26:09.576063 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:26:09.577195 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:26:09.577195 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:26:09.577195 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 25 04:26:09.577195 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 25 04:26:09.577195 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 25 04:26:09.577195 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 25 04:26:09.577195 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 25 04:26:09.577195 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 25 04:26:09.577195 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 25 04:26:09.577195 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:26:09.577195 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 624/1239] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:26:09 2022] GET /identity/v3/users/placement => generated 87 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:26:09.582665 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35d0393e-281e-4467-ba64-d108c843211f None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:26:09.687342 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35d0393e-281e-4467-ba64-d108c843211f None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:26:09.723235 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b2e2a82-1390-4e0c-becc-13789e1461bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:26:09.723855 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b2e2a82-1390-4e0c-becc-13789e1461bc None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:26:09.724150 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b2e2a82-1390-4e0c-becc-13789e1461bc None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:26:09.724411 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b2e2a82-1390-4e0c-becc-13789e1461bc None admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:26:09.725194 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b2e2a82-1390-4e0c-becc-13789e1461bc None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:26:09.725504 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b2e2a82-1390-4e0c-becc-13789e1461bc None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:26:09.726931 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b2e2a82-1390-4e0c-becc-13789e1461bc None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:26:09.747244 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 616/1240] 10.222.0.25 () {56 vars in 1143 bytes} [Sun Sep 25 04:26:09 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:26:09.753436 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2573c08e-d94a-4d2b-8029-0afd929700cc None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:26:09.866083 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2573c08e-d94a-4d2b-8029-0afd929700cc None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:26:09.906187 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a9bb9a6-66d6-467c-ac13-082f8a9ebd7d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:26:09.906850 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a9bb9a6-66d6-467c-ac13-082f8a9ebd7d None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:26:09.907174 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a9bb9a6-66d6-467c-ac13-082f8a9ebd7d None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:26:09.907533 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a9bb9a6-66d6-467c-ac13-082f8a9ebd7d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:26:09.975959 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:26:09.986398 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a9bb9a6-66d6-467c-ac13-082f8a9ebd7d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:26:09.988198 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a9bb9a6-66d6-467c-ac13-082f8a9ebd7d None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:26:10.095335 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 625/1241] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Sep 25 04:26:09 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 343 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 25 04:26:10.101241 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b2e2a82-1390-4e0c-becc-13789e1461bc None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:26:10.134419 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:26:10.224815 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b2e2a82-1390-4e0c-becc-13789e1461bc None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:26:10.259762 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9bdca81-57bb-4ae3-a42e-0f74011762a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:26:10.260396 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9bdca81-57bb-4ae3-a42e-0f74011762a0 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:26:10.260698 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9bdca81-57bb-4ae3-a42e-0f74011762a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:26:10.261012 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9bdca81-57bb-4ae3-a42e-0f74011762a0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:26:10.271666 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9bdca81-57bb-4ae3-a42e-0f74011762a0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:26:10.273167 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9bdca81-57bb-4ae3-a42e-0f74011762a0 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:26:10.288952 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-f9bdca81-57bb-4ae3-a42e-0f74011762a0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:26:10.288952 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:26:10.288952 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:26:10.288952 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:26:10.288952 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:26:10.288952 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:26:10.288952 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:26:10.288952 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:26:10.288952 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:26:10.288952 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:26:10.288952 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:26:10.288952 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:26:10.288952 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 04:26:10.288952 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 04:26:10.288952 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 04:26:10.288952 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 04:26:10.289990 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:26:10.289990 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:26:10.289990 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:26:10.289990 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:26:10.289990 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:26:10.289990 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:26:10.289990 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:26:10.289990 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:26:10.289990 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:26:10.289990 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:26:10.289990 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:26:10.289990 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:26:10.289990 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:26:10.289990 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:26:10.289990 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:26:10.289990 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:26:10.289990 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 04:26:10.291153 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 04:26:10.291153 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:26:10.291153 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:26:10.291153 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 617/1242] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:26:10 2022] GET /identity/v3/domains/Default => generated 87 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 04:26:10.296512 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a9bb9a6-66d6-467c-ac13-082f8a9ebd7d None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:26:10.401272 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:26:10.408259 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a9bb9a6-66d6-467c-ac13-082f8a9ebd7d None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:26:10.442688 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a38f0f2-fc06-4e06-8e5a-4b24e8b9a709 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:26:10.443297 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a38f0f2-fc06-4e06-8e5a-4b24e8b9a709 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:26:10.443573 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a38f0f2-fc06-4e06-8e5a-4b24e8b9a709 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:26:10.443836 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a38f0f2-fc06-4e06-8e5a-4b24e8b9a709 None admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:26:10.444465 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a38f0f2-fc06-4e06-8e5a-4b24e8b9a709 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:26:10.444792 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a38f0f2-fc06-4e06-8e5a-4b24e8b9a709 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:26:10.446152 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a38f0f2-fc06-4e06-8e5a-4b24e8b9a709 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:26:10.459427 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 626/1243] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Sep 25 04:26:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:26:10.465467 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9bdca81-57bb-4ae3-a42e-0f74011762a0 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:26:10.553663 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:26:10.577291 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9bdca81-57bb-4ae3-a42e-0f74011762a0 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:26:10.614772 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df038ca2-e6e1-4199-aaca-5a8f5f001815 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:26:10.615415 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df038ca2-e6e1-4199-aaca-5a8f5f001815 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:26:10.615696 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df038ca2-e6e1-4199-aaca-5a8f5f001815 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:26:10.615965 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df038ca2-e6e1-4199-aaca-5a8f5f001815 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:26:10.624391 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df038ca2-e6e1-4199-aaca-5a8f5f001815 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:26:10.625884 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df038ca2-e6e1-4199-aaca-5a8f5f001815 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:26:10.637384 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-df038ca2-e6e1-4199-aaca-5a8f5f001815 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 04:26:10.637384 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:26:10.637384 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:26:10.637384 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:26:10.637384 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:26:10.637384 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:26:10.637384 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:26:10.637384 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:26:10.637384 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:26:10.637384 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:26:10.637384 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:26:10.637384 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:26:10.637384 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 25 04:26:10.637384 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 25 04:26:10.637384 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 25 04:26:10.638401 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 25 04:26:10.638401 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:26:10.638401 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:26:10.638401 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:26:10.638401 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:26:10.638401 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:26:10.638401 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:26:10.638401 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:26:10.638401 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:26:10.638401 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:26:10.638401 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:26:10.638401 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:26:10.638401 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:26:10.638401 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:26:10.638401 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:26:10.638401 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:26:10.638401 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:26:10.639693 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 25 04:26:10.639693 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 25 04:26:10.639693 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 25 04:26:10.639693 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 25 04:26:10.639693 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 25 04:26:10.639693 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 25 04:26:10.639693 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 04:26:10.639693 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:26:10.639693 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 618/1244] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Sep 25 04:26:10 2022] GET /identity/v3/projects/service => generated 88 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:26:10.644456 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a38f0f2-fc06-4e06-8e5a-4b24e8b9a709 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:26:10.759395 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a38f0f2-fc06-4e06-8e5a-4b24e8b9a709 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:26:10.797833 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e942b1f-cb50-4e50-85ec-e4c274c17152 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:26:10.798571 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e942b1f-cb50-4e50-85ec-e4c274c17152 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:26:10.798863 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e942b1f-cb50-4e50-85ec-e4c274c17152 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:26:10.799140 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e942b1f-cb50-4e50-85ec-e4c274c17152 None admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:26:10.799899 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e942b1f-cb50-4e50-85ec-e4c274c17152 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:26:10.800209 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e942b1f-cb50-4e50-85ec-e4c274c17152 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:26:10.801664 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e942b1f-cb50-4e50-85ec-e4c274c17152 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:26:10.815825 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 627/1245] 10.222.0.25 () {56 vars in 1148 bytes} [Sun Sep 25 04:26:10 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:26:10.825038 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df038ca2-e6e1-4199-aaca-5a8f5f001815 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:26:10.958771 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df038ca2-e6e1-4199-aaca-5a8f5f001815 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:26:10.997541 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71dbc5cb-69f4-4103-944e-d0b7d08dfbbe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:26:10.998231 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71dbc5cb-69f4-4103-944e-d0b7d08dfbbe None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:26:10.998513 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71dbc5cb-69f4-4103-944e-d0b7d08dfbbe None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:26:10.998787 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71dbc5cb-69f4-4103-944e-d0b7d08dfbbe None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:26:10.999566 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71dbc5cb-69f4-4103-944e-d0b7d08dfbbe None admin] RBAC: Adding query filter params (role.id=9b44520f482c4423bd10568d0f86718c, scope.project.id=056ca1b1a9974a7581550de0d9217823, user.id=676d04ef5a984edcaad63d9156568660) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:26:10.999984 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71dbc5cb-69f4-4103-944e-d0b7d08dfbbe None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:26:11.001720 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71dbc5cb-69f4-4103-944e-d0b7d08dfbbe None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:26:11.007741 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 619/1246] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Sep 25 04:26:10 2022] GET /identity/v3/role_assignments?user.id=676d04ef5a984edcaad63d9156568660&scope.project.id=056ca1b1a9974a7581550de0d9217823&role.id=9b44520f482c4423bd10568d0f86718c => generated 608 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:26:13.470784 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77fa559d-a7a7-4dca-9101-8ce46c97bba6 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:26:13.471340 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77fa559d-a7a7-4dca-9101-8ce46c97bba6 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:26:13.471340 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77fa559d-a7a7-4dca-9101-8ce46c97bba6 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:26:13.472659 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 628/1247] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:26:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:26:13.478623 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef3bcc7-fbf2-4f61-ae5d-1d6df1519d0e None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:26:13.479157 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef3bcc7-fbf2-4f61-ae5d-1d6df1519d0e None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:26:13.479461 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef3bcc7-fbf2-4f61-ae5d-1d6df1519d0e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:26:13.487103 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:26:13.501688 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-eef3bcc7-fbf2-4f61-ae5d-1d6df1519d0e None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:26:13.503021 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-eef3bcc7-fbf2-4f61-ae5d-1d6df1519d0e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:26:13.581709 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 620/1248] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:26:13 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 04:26:13.859213 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77fa559d-a7a7-4dca-9101-8ce46c97bba6 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:26:13.879170 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:26:13.967721 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77fa559d-a7a7-4dca-9101-8ce46c97bba6 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:26:14.004170 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fbf8401-2b70-4e57-8f3c-2728c21c80f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:26:14.005203 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fbf8401-2b70-4e57-8f3c-2728c21c80f2 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:26:14.005463 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fbf8401-2b70-4e57-8f3c-2728c21c80f2 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:26:14.005770 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fbf8401-2b70-4e57-8f3c-2728c21c80f2 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:26:14.016664 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fbf8401-2b70-4e57-8f3c-2728c21c80f2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 04:26:14.021542 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-0fbf8401-2b70-4e57-8f3c-2728c21c80f2 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 04:26:14.021542 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:26:14.021542 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:26:14.021542 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:26:14.021542 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:26:14.021542 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:26:14.021542 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:26:14.021542 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:26:14.021542 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:26:14.021542 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:26:14.021542 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:26:14.021542 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:26:14.021542 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 04:26:14.021542 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 04:26:14.021542 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 04:26:14.021542 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 04:26:14.022712 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 04:26:14.022712 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 04:26:14.022712 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 04:26:14.022712 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 04:26:14.022712 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:26:14.022712 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:26:14.022712 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:26:14.022712 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:26:14.022712 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:26:14.022712 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:26:14.022712 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:26:14.022712 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:26:14.022712 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:26:14.022712 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:26:14.022712 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:26:14.022712 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:26:14.023916 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:26:14.023916 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:26:14.023916 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:26:14.023916 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:26:14.023916 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 04:26:14.023916 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 04:26:14.023916 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 04:26:14.023916 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 04:26:14.023916 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 04:26:14.023916 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 04:26:14.023916 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 04:26:14.023916 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:26:14.023916 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 629/1249] 10.222.0.25 () {56 vars in 1096 bytes} [Sun Sep 25 04:26:13 2022] GET /identity/v3/roles/admin => generated 83 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:26:14.029697 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eef3bcc7-fbf2-4f61-ae5d-1d6df1519d0e None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:26:14.143126 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eef3bcc7-fbf2-4f61-ae5d-1d6df1519d0e None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:26:14.177757 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:26:14.183443 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2be8925-26a1-45a9-9b9f-61a23784a918 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:26:14.184099 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2be8925-26a1-45a9-9b9f-61a23784a918 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:26:14.184412 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2be8925-26a1-45a9-9b9f-61a23784a918 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:26:14.184734 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2be8925-26a1-45a9-9b9f-61a23784a918 None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:26:14.185439 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2be8925-26a1-45a9-9b9f-61a23784a918 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:26:14.185718 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2be8925-26a1-45a9-9b9f-61a23784a918 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:26:14.187204 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2be8925-26a1-45a9-9b9f-61a23784a918 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:26:14.196178 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 621/1250] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Sep 25 04:26:14 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:26:14.202907 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fbf8401-2b70-4e57-8f3c-2728c21c80f2 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:26:14.287263 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:26:14.312146 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fbf8401-2b70-4e57-8f3c-2728c21c80f2 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:26:14.349869 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f2c6478-c982-4c88-b8be-d8095d6191aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:26:14.350512 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f2c6478-c982-4c88-b8be-d8095d6191aa None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:26:14.350790 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f2c6478-c982-4c88-b8be-d8095d6191aa None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:26:14.351058 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f2c6478-c982-4c88-b8be-d8095d6191aa None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:26:14.362275 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f2c6478-c982-4c88-b8be-d8095d6191aa None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:26:14.363759 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f2c6478-c982-4c88-b8be-d8095d6191aa None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:26:14.379921 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-4f2c6478-c982-4c88-b8be-d8095d6191aa None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:26:14.379921 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:26:14.379921 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:26:14.379921 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:26:14.379921 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:26:14.379921 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:26:14.379921 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:26:14.379921 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:26:14.379921 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:26:14.379921 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:26:14.379921 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:26:14.379921 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:26:14.379921 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 04:26:14.379921 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 04:26:14.379921 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 04:26:14.379921 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 04:26:14.381090 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:26:14.381090 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:26:14.381090 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:26:14.381090 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:26:14.381090 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:26:14.381090 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:26:14.381090 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:26:14.381090 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:26:14.381090 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:26:14.381090 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:26:14.381090 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:26:14.381090 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:26:14.381090 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:26:14.381090 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:26:14.381090 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:26:14.381090 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:26:14.381090 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 04:26:14.382088 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 04:26:14.382088 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:26:14.382088 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:26:14.382088 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 630/1251] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:26:14 2022] GET /identity/v3/domains/Default => generated 87 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:26:14.388038 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2be8925-26a1-45a9-9b9f-61a23784a918 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:26:14.493706 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2be8925-26a1-45a9-9b9f-61a23784a918 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:26:14.529380 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e017a2ea-3baf-44ba-af3b-39f821d34a57 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:26:14.530009 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e017a2ea-3baf-44ba-af3b-39f821d34a57 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:26:14.530291 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e017a2ea-3baf-44ba-af3b-39f821d34a57 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:26:14.530567 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e017a2ea-3baf-44ba-af3b-39f821d34a57 None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:26:14.531225 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e017a2ea-3baf-44ba-af3b-39f821d34a57 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:26:14.531532 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e017a2ea-3baf-44ba-af3b-39f821d34a57 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:26:14.532975 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e017a2ea-3baf-44ba-af3b-39f821d34a57 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:26:14.550712 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 622/1252] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Sep 25 04:26:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:26:14.552040 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f2c6478-c982-4c88-b8be-d8095d6191aa None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:26:14.659350 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f2c6478-c982-4c88-b8be-d8095d6191aa None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:26:14.686307 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:26:14.694379 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7a3ea5d-dcaa-40d1-a0f2-a9194d958c53 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:26:14.695006 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7a3ea5d-dcaa-40d1-a0f2-a9194d958c53 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:26:14.695306 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7a3ea5d-dcaa-40d1-a0f2-a9194d958c53 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:26:14.695570 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7a3ea5d-dcaa-40d1-a0f2-a9194d958c53 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:26:14.704424 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7a3ea5d-dcaa-40d1-a0f2-a9194d958c53 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:26:14.705899 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7a3ea5d-dcaa-40d1-a0f2-a9194d958c53 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:26:14.717752 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-f7a3ea5d-dcaa-40d1-a0f2-a9194d958c53 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 25 04:26:14.717752 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:26:14.717752 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:26:14.717752 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:26:14.717752 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:26:14.717752 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:26:14.717752 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:26:14.717752 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:26:14.717752 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:26:14.717752 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:26:14.717752 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:26:14.717752 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:26:14.717752 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 25 04:26:14.717752 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 25 04:26:14.717752 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 25 04:26:14.717752 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 25 04:26:14.718786 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:26:14.718786 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:26:14.718786 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 25 04:26:14.718786 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 04:26:14.718786 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 25 04:26:14.718786 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 04:26:14.718786 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:26:14.718786 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:26:14.718786 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:26:14.718786 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:26:14.718786 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:26:14.718786 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:26:14.718786 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:26:14.718786 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:26:14.718786 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:26:14.718786 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:26:14.718786 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:26:14.718786 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:26:14.720065 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:26:14.720065 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:26:14.720065 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 25 04:26:14.720065 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 25 04:26:14.720065 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 25 04:26:14.720065 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 25 04:26:14.720065 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 25 04:26:14.720065 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 25 04:26:14.720065 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 25 04:26:14.720065 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:26:14.720065 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 631/1253] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:26:14 2022] GET /identity/v3/users/placement => generated 87 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:26:14.724735 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e017a2ea-3baf-44ba-af3b-39f821d34a57 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:26:14.782343 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:26:14.829155 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e017a2ea-3baf-44ba-af3b-39f821d34a57 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:26:14.865408 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4085ff0f-300e-4d52-bc54-5b20a4cb2a80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:26:14.866020 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4085ff0f-300e-4d52-bc54-5b20a4cb2a80 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:26:14.866300 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4085ff0f-300e-4d52-bc54-5b20a4cb2a80 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:26:14.866567 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4085ff0f-300e-4d52-bc54-5b20a4cb2a80 None admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:26:14.867289 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4085ff0f-300e-4d52-bc54-5b20a4cb2a80 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:26:14.867592 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4085ff0f-300e-4d52-bc54-5b20a4cb2a80 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:26:14.869059 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4085ff0f-300e-4d52-bc54-5b20a4cb2a80 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:26:14.886607 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 623/1254] 10.222.0.25 () {56 vars in 1143 bytes} [Sun Sep 25 04:26:14 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:26:14.892480 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7a3ea5d-dcaa-40d1-a0f2-a9194d958c53 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:26:14.989005 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7a3ea5d-dcaa-40d1-a0f2-a9194d958c53 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:26:15.021512 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2541a95-4d81-4aa5-ba89-c81c3df295a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:26:15.022101 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2541a95-4d81-4aa5-ba89-c81c3df295a9 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:26:15.022376 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2541a95-4d81-4aa5-ba89-c81c3df295a9 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:26:15.022641 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2541a95-4d81-4aa5-ba89-c81c3df295a9 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:26:15.081986 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:26:15.094870 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2541a95-4d81-4aa5-ba89-c81c3df295a9 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:26:15.096298 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2541a95-4d81-4aa5-ba89-c81c3df295a9 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:26:15.203951 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 632/1255] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Sep 25 04:26:14 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 312 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 25 04:26:15.209738 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4085ff0f-300e-4d52-bc54-5b20a4cb2a80 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:26:15.305225 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4085ff0f-300e-4d52-bc54-5b20a4cb2a80 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:26:15.318944 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:26:15.340479 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a874c29-2755-419a-8216-79cb4b3f13be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:26:15.341069 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a874c29-2755-419a-8216-79cb4b3f13be None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:26:15.341350 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a874c29-2755-419a-8216-79cb4b3f13be None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:26:15.341621 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a874c29-2755-419a-8216-79cb4b3f13be None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:26:15.351275 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a874c29-2755-419a-8216-79cb4b3f13be None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:26:15.352733 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a874c29-2755-419a-8216-79cb4b3f13be None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:26:15.366669 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-5a874c29-2755-419a-8216-79cb4b3f13be None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:26:15.366669 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:26:15.366669 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:26:15.366669 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:26:15.366669 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:26:15.366669 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:26:15.366669 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:26:15.366669 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:26:15.366669 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:26:15.366669 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:26:15.366669 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:26:15.366669 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:26:15.366669 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 04:26:15.366669 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 04:26:15.366669 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 04:26:15.366669 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 04:26:15.367802 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:26:15.367802 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:26:15.367802 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:26:15.367802 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:26:15.367802 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:26:15.367802 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:26:15.367802 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:26:15.367802 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:26:15.367802 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:26:15.367802 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:26:15.367802 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:26:15.367802 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:26:15.367802 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:26:15.367802 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:26:15.367802 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:26:15.367802 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:26:15.367802 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 04:26:15.369005 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 04:26:15.369005 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:26:15.369005 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:26:15.369005 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 624/1256] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:26:15 2022] GET /identity/v3/domains/Default => generated 87 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:26:15.374958 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2541a95-4d81-4aa5-ba89-c81c3df295a9 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:26:15.460215 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:26:15.472830 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2541a95-4d81-4aa5-ba89-c81c3df295a9 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:26:15.506251 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ebda145-8b12-4f6c-a842-728c9743c965 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:26:15.506853 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ebda145-8b12-4f6c-a842-728c9743c965 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:26:15.507132 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ebda145-8b12-4f6c-a842-728c9743c965 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:26:15.507404 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ebda145-8b12-4f6c-a842-728c9743c965 None admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:26:15.508034 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ebda145-8b12-4f6c-a842-728c9743c965 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:26:15.508335 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ebda145-8b12-4f6c-a842-728c9743c965 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:26:15.509716 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ebda145-8b12-4f6c-a842-728c9743c965 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:26:15.521067 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 633/1257] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Sep 25 04:26:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:26:15.527295 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a874c29-2755-419a-8216-79cb4b3f13be None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:26:15.625631 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a874c29-2755-419a-8216-79cb4b3f13be None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:26:15.658553 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6bbc160-8d75-4ca8-bbce-4a5c6060c27c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:26:15.659168 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6bbc160-8d75-4ca8-bbce-4a5c6060c27c None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:26:15.659447 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6bbc160-8d75-4ca8-bbce-4a5c6060c27c None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:26:15.659719 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6bbc160-8d75-4ca8-bbce-4a5c6060c27c None admin] PATH_INFO: `/v3/projects/service` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:26:15.666037 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6bbc160-8d75-4ca8-bbce-4a5c6060c27c None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:26:15.667510 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6bbc160-8d75-4ca8-bbce-4a5c6060c27c None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:26:15.678403 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-c6bbc160-8d75-4ca8-bbce-4a5c6060c27c None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 04:26:15.678403 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:26:15.678403 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:26:15.678403 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:26:15.678403 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:26:15.678403 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:26:15.678403 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:26:15.678403 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:26:15.678403 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:26:15.678403 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:26:15.678403 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:26:15.678403 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:26:15.678403 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 25 04:26:15.678403 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 25 04:26:15.678403 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 25 04:26:15.679491 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 25 04:26:15.679491 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:26:15.679491 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:26:15.679491 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:26:15.679491 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:26:15.679491 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:26:15.679491 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:26:15.679491 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:26:15.679491 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:26:15.679491 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:26:15.679491 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:26:15.679491 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:26:15.679491 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:26:15.679491 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:26:15.679491 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:26:15.679491 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:26:15.679491 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:26:15.680446 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 25 04:26:15.680446 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 25 04:26:15.680446 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 25 04:26:15.680446 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 25 04:26:15.680446 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 25 04:26:15.680446 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 25 04:26:15.680446 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 04:26:15.680446 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:26:15.680446 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 625/1258] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Sep 25 04:26:15 2022] GET /identity/v3/projects/service => generated 88 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:26:15.685451 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ebda145-8b12-4f6c-a842-728c9743c965 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:26:15.781063 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ebda145-8b12-4f6c-a842-728c9743c965 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:26:15.812817 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e68b6ca5-a23d-45f2-a270-8d3204c263b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:26:15.813406 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e68b6ca5-a23d-45f2-a270-8d3204c263b8 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:26:15.813683 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e68b6ca5-a23d-45f2-a270-8d3204c263b8 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:26:15.813953 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e68b6ca5-a23d-45f2-a270-8d3204c263b8 None admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:26:15.814631 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e68b6ca5-a23d-45f2-a270-8d3204c263b8 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:26:15.814937 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e68b6ca5-a23d-45f2-a270-8d3204c263b8 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:26:15.816344 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e68b6ca5-a23d-45f2-a270-8d3204c263b8 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:26:15.828517 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 634/1259] 10.222.0.25 () {56 vars in 1148 bytes} [Sun Sep 25 04:26:15 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:26:15.834435 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6bbc160-8d75-4ca8-bbce-4a5c6060c27c None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:26:15.860747 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:26:15.932588 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6bbc160-8d75-4ca8-bbce-4a5c6060c27c None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:26:15.966225 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-679ff9f6-b663-4fe3-afd6-007360cf4648 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:26:15.966819 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-679ff9f6-b663-4fe3-afd6-007360cf4648 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:26:15.967099 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-679ff9f6-b663-4fe3-afd6-007360cf4648 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:26:15.967371 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-679ff9f6-b663-4fe3-afd6-007360cf4648 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:26:15.968082 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-679ff9f6-b663-4fe3-afd6-007360cf4648 None admin] RBAC: Adding query filter params (role.id=c51fe7c5ad804040879acfb54269455f, scope.project.id=056ca1b1a9974a7581550de0d9217823, user.id=676d04ef5a984edcaad63d9156568660) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:26:15.968389 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-679ff9f6-b663-4fe3-afd6-007360cf4648 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:26:15.969840 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-679ff9f6-b663-4fe3-afd6-007360cf4648 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:26:15.974976 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 626/1260] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Sep 25 04:26:15 2022] GET /identity/v3/role_assignments?user.id=676d04ef5a984edcaad63d9156568660&scope.project.id=056ca1b1a9974a7581550de0d9217823&role.id=c51fe7c5ad804040879acfb54269455f => generated 259 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:26:18.098005 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a93b94b9-4874-48ef-a084-2640ad879f57 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:26:18.098519 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a93b94b9-4874-48ef-a084-2640ad879f57 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:26:18.098613 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a93b94b9-4874-48ef-a084-2640ad879f57 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:26:18.099889 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 635/1261] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:26:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:26:18.105456 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e47c27-fe41-4132-ace9-6772ab247936 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:26:18.105860 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e47c27-fe41-4132-ace9-6772ab247936 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:26:18.106031 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e47c27-fe41-4132-ace9-6772ab247936 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:26:18.124192 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-b6e47c27-fe41-4132-ace9-6772ab247936 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:26:18.125405 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-b6e47c27-fe41-4132-ace9-6772ab247936 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:26:18.202132 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 627/1262] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:26:18 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 04:26:18.471414 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a93b94b9-4874-48ef-a084-2640ad879f57 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:26:18.482962 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:26:18.572053 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a93b94b9-4874-48ef-a084-2640ad879f57 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:26:18.605758 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3253a3ee-50f2-4ab7-8888-3c524b7d10dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:26:18.606386 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3253a3ee-50f2-4ab7-8888-3c524b7d10dc None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:26:18.606662 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3253a3ee-50f2-4ab7-8888-3c524b7d10dc None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:26:18.606929 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3253a3ee-50f2-4ab7-8888-3c524b7d10dc None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:26:18.616664 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3253a3ee-50f2-4ab7-8888-3c524b7d10dc None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 04:26:18.621472 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-3253a3ee-50f2-4ab7-8888-3c524b7d10dc None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 04:26:18.621472 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:26:18.621472 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:26:18.621472 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:26:18.621472 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:26:18.621472 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:26:18.621472 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:26:18.621472 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:26:18.621472 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:26:18.621472 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:26:18.621472 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:26:18.621472 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:26:18.621472 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 04:26:18.621472 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 04:26:18.621472 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 04:26:18.621472 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 04:26:18.622583 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 04:26:18.622583 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 04:26:18.622583 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 04:26:18.622583 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 04:26:18.622583 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:26:18.622583 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:26:18.622583 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:26:18.622583 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:26:18.622583 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:26:18.622583 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:26:18.622583 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:26:18.622583 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:26:18.622583 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:26:18.622583 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:26:18.622583 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:26:18.622583 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:26:18.623697 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:26:18.623697 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:26:18.623697 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:26:18.623697 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:26:18.623697 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 04:26:18.623697 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 04:26:18.623697 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 04:26:18.623697 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 04:26:18.623697 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 04:26:18.623697 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 04:26:18.623697 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 04:26:18.623697 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:26:18.623697 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 636/1263] 10.222.0.25 () {56 vars in 1096 bytes} [Sun Sep 25 04:26:18 2022] GET /identity/v3/roles/admin => generated 83 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:26:18.628502 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6e47c27-fe41-4132-ace9-6772ab247936 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:26:18.647047 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:26:18.726512 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6e47c27-fe41-4132-ace9-6772ab247936 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:26:18.759486 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9985cc2e-778f-4d67-a629-9e375235f006 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:26:18.760085 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9985cc2e-778f-4d67-a629-9e375235f006 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:26:18.760365 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9985cc2e-778f-4d67-a629-9e375235f006 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:26:18.760662 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9985cc2e-778f-4d67-a629-9e375235f006 None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:26:18.762186 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9985cc2e-778f-4d67-a629-9e375235f006 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:26:18.762494 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9985cc2e-778f-4d67-a629-9e375235f006 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:26:18.763861 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9985cc2e-778f-4d67-a629-9e375235f006 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:26:18.771615 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 628/1264] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Sep 25 04:26:18 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:26:18.777786 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3253a3ee-50f2-4ab7-8888-3c524b7d10dc None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:26:18.842715 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:26:18.874593 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3253a3ee-50f2-4ab7-8888-3c524b7d10dc None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:26:18.907356 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92a076b8-335e-45fc-8051-9f9ecd393bee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:26:18.907977 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92a076b8-335e-45fc-8051-9f9ecd393bee None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:26:18.908255 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92a076b8-335e-45fc-8051-9f9ecd393bee None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:26:18.908525 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92a076b8-335e-45fc-8051-9f9ecd393bee None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:26:18.918164 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92a076b8-335e-45fc-8051-9f9ecd393bee None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:26:18.919595 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92a076b8-335e-45fc-8051-9f9ecd393bee None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:26:18.934144 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-92a076b8-335e-45fc-8051-9f9ecd393bee None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:26:18.934144 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:26:18.934144 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:26:18.934144 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:26:18.934144 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:26:18.934144 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:26:18.934144 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:26:18.934144 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:26:18.934144 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:26:18.934144 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:26:18.934144 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:26:18.934144 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:26:18.934144 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 04:26:18.934144 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 04:26:18.934144 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 04:26:18.934144 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 04:26:18.935243 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:26:18.935243 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:26:18.935243 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:26:18.935243 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:26:18.935243 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:26:18.935243 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:26:18.935243 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:26:18.935243 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:26:18.935243 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:26:18.935243 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:26:18.935243 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:26:18.935243 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:26:18.935243 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:26:18.935243 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:26:18.935243 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:26:18.935243 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:26:18.935243 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 04:26:18.936254 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 04:26:18.936254 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:26:18.936254 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:26:18.936254 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 637/1265] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:26:18 2022] GET /identity/v3/domains/Default => generated 87 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:26:18.941164 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9985cc2e-778f-4d67-a629-9e375235f006 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:26:19.020850 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:26:19.040461 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9985cc2e-778f-4d67-a629-9e375235f006 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:26:19.072816 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5da3b57-9b01-4bcb-86b2-b1864be54f46 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:26:19.073604 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5da3b57-9b01-4bcb-86b2-b1864be54f46 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:26:19.074021 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5da3b57-9b01-4bcb-86b2-b1864be54f46 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:26:19.074302 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5da3b57-9b01-4bcb-86b2-b1864be54f46 None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:26:19.074938 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5da3b57-9b01-4bcb-86b2-b1864be54f46 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:26:19.075246 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5da3b57-9b01-4bcb-86b2-b1864be54f46 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:26:19.076605 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5da3b57-9b01-4bcb-86b2-b1864be54f46 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:26:19.088178 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 629/1266] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Sep 25 04:26:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:26:19.094383 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92a076b8-335e-45fc-8051-9f9ecd393bee None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:26:19.194220 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92a076b8-335e-45fc-8051-9f9ecd393bee None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:26:19.213161 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:26:19.227807 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1e6eafd-3a25-4df9-9e62-bc73df1be955 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:26:19.228428 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1e6eafd-3a25-4df9-9e62-bc73df1be955 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:26:19.228731 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1e6eafd-3a25-4df9-9e62-bc73df1be955 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:26:19.229002 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1e6eafd-3a25-4df9-9e62-bc73df1be955 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:26:19.236963 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1e6eafd-3a25-4df9-9e62-bc73df1be955 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:26:19.238389 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1e6eafd-3a25-4df9-9e62-bc73df1be955 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:26:19.249297 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-e1e6eafd-3a25-4df9-9e62-bc73df1be955 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 25 04:26:19.249297 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:26:19.249297 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:26:19.249297 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:26:19.249297 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:26:19.249297 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:26:19.249297 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:26:19.249297 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:26:19.249297 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:26:19.249297 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:26:19.249297 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:26:19.249297 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:26:19.249297 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 25 04:26:19.249297 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 25 04:26:19.249297 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 25 04:26:19.249297 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 25 04:26:19.250418 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:26:19.250418 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:26:19.250418 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 25 04:26:19.250418 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 04:26:19.250418 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 25 04:26:19.250418 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 04:26:19.250418 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:26:19.250418 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:26:19.250418 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:26:19.250418 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:26:19.250418 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:26:19.250418 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:26:19.250418 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:26:19.250418 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:26:19.250418 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:26:19.250418 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:26:19.250418 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:26:19.250418 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:26:19.251469 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:26:19.251469 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:26:19.251469 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 25 04:26:19.251469 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 25 04:26:19.251469 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 25 04:26:19.251469 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 25 04:26:19.251469 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 25 04:26:19.251469 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 25 04:26:19.251469 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 25 04:26:19.251469 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:26:19.251469 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 638/1267] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:26:19 2022] GET /identity/v3/users/placement => generated 87 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:26:19.256185 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5da3b57-9b01-4bcb-86b2-b1864be54f46 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:26:19.353219 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5da3b57-9b01-4bcb-86b2-b1864be54f46 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:26:19.385371 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7bb8758-c545-4390-817c-523d6ddfeae5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:26:19.386530 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7bb8758-c545-4390-817c-523d6ddfeae5 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:26:19.386806 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7bb8758-c545-4390-817c-523d6ddfeae5 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:26:19.387078 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7bb8758-c545-4390-817c-523d6ddfeae5 None admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:26:19.387796 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7bb8758-c545-4390-817c-523d6ddfeae5 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:26:19.388101 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7bb8758-c545-4390-817c-523d6ddfeae5 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:26:19.389558 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7bb8758-c545-4390-817c-523d6ddfeae5 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:26:19.397847 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:26:19.407021 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 630/1268] 10.222.0.25 () {56 vars in 1143 bytes} [Sun Sep 25 04:26:19 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:26:19.412840 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1e6eafd-3a25-4df9-9e62-bc73df1be955 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:26:19.508562 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1e6eafd-3a25-4df9-9e62-bc73df1be955 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:26:19.541449 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-add5e2c1-3657-4954-a564-2bc7864df947 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:26:19.542116 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add5e2c1-3657-4954-a564-2bc7864df947 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:26:19.542611 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add5e2c1-3657-4954-a564-2bc7864df947 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:26:19.542890 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add5e2c1-3657-4954-a564-2bc7864df947 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:26:19.552874 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-add5e2c1-3657-4954-a564-2bc7864df947 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:26:19.554305 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-add5e2c1-3657-4954-a564-2bc7864df947 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:26:19.568076 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-add5e2c1-3657-4954-a564-2bc7864df947 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:26:19.568076 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:26:19.568076 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:26:19.568076 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:26:19.568076 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:26:19.568076 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:26:19.568076 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:26:19.568076 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:26:19.568076 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:26:19.568076 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:26:19.568076 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:26:19.568076 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:26:19.568076 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 04:26:19.568076 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 04:26:19.568076 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 04:26:19.568076 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 04:26:19.569270 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:26:19.569270 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:26:19.569270 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:26:19.569270 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:26:19.569270 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:26:19.569270 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:26:19.569270 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:26:19.569270 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:26:19.569270 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:26:19.569270 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:26:19.569270 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:26:19.569270 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:26:19.569270 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:26:19.569270 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:26:19.569270 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:26:19.569270 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:26:19.569270 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 04:26:19.570498 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 04:26:19.570498 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:26:19.570498 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:26:19.570498 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 639/1269] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:26:19 2022] GET /identity/v3/domains/Default => generated 87 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:26:19.575451 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7bb8758-c545-4390-817c-523d6ddfeae5 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:26:19.672146 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7bb8758-c545-4390-817c-523d6ddfeae5 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:26:19.705902 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d1a77a6-23ba-4a30-b4d2-75590fe88342 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:26:19.706496 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d1a77a6-23ba-4a30-b4d2-75590fe88342 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:26:19.706774 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d1a77a6-23ba-4a30-b4d2-75590fe88342 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:26:19.707051 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d1a77a6-23ba-4a30-b4d2-75590fe88342 None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:26:19.707705 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d1a77a6-23ba-4a30-b4d2-75590fe88342 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:26:19.708006 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d1a77a6-23ba-4a30-b4d2-75590fe88342 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:26:19.709389 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d1a77a6-23ba-4a30-b4d2-75590fe88342 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:26:19.720222 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 631/1270] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Sep 25 04:26:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:26:19.725887 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-add5e2c1-3657-4954-a564-2bc7864df947 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:26:19.757312 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:26:19.822147 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-add5e2c1-3657-4954-a564-2bc7864df947 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:26:19.855391 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-129d5250-ad33-47a8-bf2f-2c60757307b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:26:19.856005 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-129d5250-ad33-47a8-bf2f-2c60757307b7 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:26:19.856280 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-129d5250-ad33-47a8-bf2f-2c60757307b7 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:26:19.856544 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-129d5250-ad33-47a8-bf2f-2c60757307b7 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:26:19.862915 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-129d5250-ad33-47a8-bf2f-2c60757307b7 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:26:19.864333 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-129d5250-ad33-47a8-bf2f-2c60757307b7 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:26:19.876083 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-129d5250-ad33-47a8-bf2f-2c60757307b7 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 04:26:19.876083 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:26:19.876083 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:26:19.876083 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:26:19.876083 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:26:19.876083 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:26:19.876083 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:26:19.876083 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:26:19.876083 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:26:19.876083 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:26:19.876083 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:26:19.876083 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:26:19.876083 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 25 04:26:19.876083 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 25 04:26:19.876083 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 25 04:26:19.877024 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 25 04:26:19.877024 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:26:19.877024 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:26:19.877024 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:26:19.877024 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:26:19.877024 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:26:19.877024 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:26:19.877024 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:26:19.877024 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:26:19.877024 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:26:19.877024 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:26:19.877024 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:26:19.877024 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:26:19.877024 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:26:19.877024 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:26:19.877024 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:26:19.877024 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:26:19.878031 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 25 04:26:19.878031 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 25 04:26:19.878031 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 25 04:26:19.878031 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 25 04:26:19.878031 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 25 04:26:19.878031 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 25 04:26:19.878031 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 04:26:19.878031 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:26:19.878031 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 640/1271] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Sep 25 04:26:19 2022] GET /identity/v3/projects/service => generated 88 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:26:19.883042 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d1a77a6-23ba-4a30-b4d2-75590fe88342 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:26:19.940515 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:26:19.981387 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d1a77a6-23ba-4a30-b4d2-75590fe88342 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:26:20.014209 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7b37a57-640a-439f-b698-25e54e7e1e90 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:26:20.014808 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7b37a57-640a-439f-b698-25e54e7e1e90 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:26:20.015086 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7b37a57-640a-439f-b698-25e54e7e1e90 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:26:20.015357 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7b37a57-640a-439f-b698-25e54e7e1e90 None admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:26:20.016037 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7b37a57-640a-439f-b698-25e54e7e1e90 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:26:20.016343 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7b37a57-640a-439f-b698-25e54e7e1e90 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:26:20.017781 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7b37a57-640a-439f-b698-25e54e7e1e90 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:26:20.030052 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 632/1272] 10.222.0.25 () {56 vars in 1148 bytes} [Sun Sep 25 04:26:19 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:26:20.036030 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-129d5250-ad33-47a8-bf2f-2c60757307b7 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:26:20.127197 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:26:20.137953 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-129d5250-ad33-47a8-bf2f-2c60757307b7 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:26:20.170912 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c10d6c5d-99dd-47d8-953f-34d973dd2222 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:26:20.171557 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c10d6c5d-99dd-47d8-953f-34d973dd2222 None admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:26:20.171833 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c10d6c5d-99dd-47d8-953f-34d973dd2222 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:26:20.172098 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c10d6c5d-99dd-47d8-953f-34d973dd2222 None admin] PATH_INFO: `/v3/projects/056ca1b1a9974a7581550de0d9217823/users/676d04ef5a984edcaad63d9156568660/roles/c51fe7c5ad804040879acfb54269455f` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:26:20.173977 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c10d6c5d-99dd-47d8-953f-34d973dd2222 None admin] RBAC: Authorizing `identity:create_grant(project_id=056ca1b1a9974a7581550de0d9217823, user_id=676d04ef5a984edcaad63d9156568660, role_id=c51fe7c5ad804040879acfb54269455f)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:26:20.175387 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c10d6c5d-99dd-47d8-953f-34d973dd2222 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:26:20.187661 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 641/1273] 10.222.0.25 () {58 vars in 1439 bytes} [Sun Sep 25 04:26:20 2022] PUT /identity/v3/projects/056ca1b1a9974a7581550de0d9217823/users/676d04ef5a984edcaad63d9156568660/roles/c51fe7c5ad804040879acfb54269455f => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:26:22.300185 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b971fc43-7e1b-4b19-893f-25f226c21d88 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:26:22.300840 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b971fc43-7e1b-4b19-893f-25f226c21d88 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:26:22.300840 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b971fc43-7e1b-4b19-893f-25f226c21d88 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:26:22.302109 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 633/1274] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:26:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:26:22.307213 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d0ae404-fc3e-4567-ac80-bba349315a71 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:26:22.307673 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d0ae404-fc3e-4567-ac80-bba349315a71 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:26:22.307902 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d0ae404-fc3e-4567-ac80-bba349315a71 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:26:22.326296 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-4d0ae404-fc3e-4567-ac80-bba349315a71 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:26:22.327463 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-4d0ae404-fc3e-4567-ac80-bba349315a71 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:26:22.391679 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 642/1275] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:26:22 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 04:26:22.672716 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b971fc43-7e1b-4b19-893f-25f226c21d88 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:26:22.772650 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b971fc43-7e1b-4b19-893f-25f226c21d88 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:26:22.789583 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:26:22.806746 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0381d14-0890-47c7-b796-e215e44cb592 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:26:22.807365 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0381d14-0890-47c7-b796-e215e44cb592 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:26:22.807667 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0381d14-0890-47c7-b796-e215e44cb592 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:26:22.807934 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0381d14-0890-47c7-b796-e215e44cb592 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:26:22.817642 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0381d14-0890-47c7-b796-e215e44cb592 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 25 04:26:22.822353 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-a0381d14-0890-47c7-b796-e215e44cb592 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 04:26:22.822353 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:26:22.822353 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:26:22.822353 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:26:22.822353 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:26:22.822353 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:26:22.822353 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:26:22.822353 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:26:22.822353 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:26:22.822353 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:26:22.822353 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:26:22.822353 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:26:22.822353 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 25 04:26:22.822353 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 25 04:26:22.822353 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 25 04:26:22.822353 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 25 04:26:22.823522 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 25 04:26:22.823522 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 25 04:26:22.823522 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 25 04:26:22.823522 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 25 04:26:22.823522 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:26:22.823522 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:26:22.823522 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:26:22.823522 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:26:22.823522 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:26:22.823522 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:26:22.823522 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:26:22.823522 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:26:22.823522 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:26:22.823522 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:26:22.823522 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:26:22.823522 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:26:22.824733 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:26:22.824733 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:26:22.824733 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:26:22.824733 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:26:22.824733 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 25 04:26:22.824733 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 25 04:26:22.824733 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 25 04:26:22.824733 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 25 04:26:22.824733 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 25 04:26:22.824733 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 25 04:26:22.824733 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 25 04:26:22.824733 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:26:22.824733 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 634/1276] 10.222.0.25 () {56 vars in 1096 bytes} [Sun Sep 25 04:26:22 2022] GET /identity/v3/roles/admin => generated 83 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:26:22.829900 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d0ae404-fc3e-4567-ac80-bba349315a71 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:26:22.906886 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:26:22.925171 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d0ae404-fc3e-4567-ac80-bba349315a71 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:26:22.957502 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4691ab0-b10b-4b72-a5ba-2b5050a25b9c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:26:22.958096 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4691ab0-b10b-4b72-a5ba-2b5050a25b9c None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:26:22.958370 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4691ab0-b10b-4b72-a5ba-2b5050a25b9c None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:26:22.958636 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4691ab0-b10b-4b72-a5ba-2b5050a25b9c None admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:26:22.959281 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4691ab0-b10b-4b72-a5ba-2b5050a25b9c None admin] RBAC: Adding query filter params (name=admin) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:26:22.959578 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4691ab0-b10b-4b72-a5ba-2b5050a25b9c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:26:22.960949 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4691ab0-b10b-4b72-a5ba-2b5050a25b9c None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:26:22.969656 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 643/1277] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Sep 25 04:26:22 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:26:22.975399 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0381d14-0890-47c7-b796-e215e44cb592 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:26:23.071601 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0381d14-0890-47c7-b796-e215e44cb592 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:26:23.105495 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-202df179-6ae7-486d-a98b-31b982445a79 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:26:23.106114 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-202df179-6ae7-486d-a98b-31b982445a79 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:26:23.106397 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-202df179-6ae7-486d-a98b-31b982445a79 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:26:23.106670 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-202df179-6ae7-486d-a98b-31b982445a79 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:26:23.117230 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-202df179-6ae7-486d-a98b-31b982445a79 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:26:23.118943 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-202df179-6ae7-486d-a98b-31b982445a79 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:26:23.136254 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-202df179-6ae7-486d-a98b-31b982445a79 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:26:23.136254 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:26:23.136254 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:26:23.136254 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:26:23.136254 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:26:23.136254 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:26:23.136254 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:26:23.136254 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:26:23.136254 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:26:23.136254 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:26:23.136254 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:26:23.136254 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:26:23.136254 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 04:26:23.136254 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 04:26:23.136254 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 04:26:23.136254 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 04:26:23.137878 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:26:23.137878 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:26:23.137878 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:26:23.137878 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:26:23.137878 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:26:23.137878 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:26:23.137878 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:26:23.137878 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:26:23.137878 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:26:23.137878 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:26:23.137878 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:26:23.137878 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:26:23.137878 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:26:23.137878 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:26:23.137878 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:26:23.137878 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:26:23.137878 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 04:26:23.138884 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 04:26:23.138884 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:26:23.138884 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:26:23.138884 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 635/1278] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:26:22 2022] GET /identity/v3/domains/Default => generated 87 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:26:23.143728 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4691ab0-b10b-4b72-a5ba-2b5050a25b9c None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:26:23.249162 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4691ab0-b10b-4b72-a5ba-2b5050a25b9c None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:26:23.281799 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbd992f3-d5d9-4d6c-94a3-49208e8f9a00 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:26:23.282397 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbd992f3-d5d9-4d6c-94a3-49208e8f9a00 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:26:23.282678 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbd992f3-d5d9-4d6c-94a3-49208e8f9a00 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:26:23.282950 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbd992f3-d5d9-4d6c-94a3-49208e8f9a00 None admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:26:23.283589 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbd992f3-d5d9-4d6c-94a3-49208e8f9a00 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:26:23.283892 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbd992f3-d5d9-4d6c-94a3-49208e8f9a00 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:26:23.285276 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbd992f3-d5d9-4d6c-94a3-49208e8f9a00 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:26:23.296439 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 644/1279] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Sep 25 04:26:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:26:23.302133 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-202df179-6ae7-486d-a98b-31b982445a79 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:26:23.332852 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:26:23.400439 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-202df179-6ae7-486d-a98b-31b982445a79 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:26:23.433305 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcb9f266-431c-46ba-9926-608bb5e40926 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:26:23.434086 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcb9f266-431c-46ba-9926-608bb5e40926 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:26:23.434367 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcb9f266-431c-46ba-9926-608bb5e40926 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:26:23.434638 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcb9f266-431c-46ba-9926-608bb5e40926 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:26:23.441768 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcb9f266-431c-46ba-9926-608bb5e40926 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:26:23.443197 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcb9f266-431c-46ba-9926-608bb5e40926 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:26:23.455212 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-bcb9f266-431c-46ba-9926-608bb5e40926 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 25 04:26:23.455212 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:26:23.455212 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:26:23.455212 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:26:23.455212 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:26:23.455212 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:26:23.455212 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:26:23.455212 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:26:23.455212 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:26:23.455212 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:26:23.455212 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:26:23.455212 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:26:23.455212 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 25 04:26:23.455212 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 25 04:26:23.455212 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 25 04:26:23.455212 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 25 04:26:23.456372 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:26:23.456372 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:26:23.456372 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 25 04:26:23.456372 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 04:26:23.456372 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 25 04:26:23.456372 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 25 04:26:23.456372 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:26:23.456372 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:26:23.456372 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:26:23.456372 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:26:23.456372 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:26:23.456372 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:26:23.456372 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:26:23.456372 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:26:23.456372 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:26:23.456372 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:26:23.456372 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:26:23.456372 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:26:23.457555 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:26:23.457555 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:26:23.457555 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 25 04:26:23.457555 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 25 04:26:23.457555 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 25 04:26:23.457555 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 25 04:26:23.457555 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 25 04:26:23.457555 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 25 04:26:23.457555 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 25 04:26:23.457555 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:26:23.457555 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 636/1280] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:26:23 2022] GET /identity/v3/users/placement => generated 87 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:26:23.462029 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbd992f3-d5d9-4d6c-94a3-49208e8f9a00 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:26:23.465875 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:26:23.561378 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbd992f3-d5d9-4d6c-94a3-49208e8f9a00 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:26:23.594992 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0d4c8db-da49-4764-bde5-25686bbcf1e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:26:23.595597 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d4c8db-da49-4764-bde5-25686bbcf1e0 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:26:23.595879 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d4c8db-da49-4764-bde5-25686bbcf1e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:26:23.596152 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d4c8db-da49-4764-bde5-25686bbcf1e0 None admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:26:23.596912 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0d4c8db-da49-4764-bde5-25686bbcf1e0 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:26:23.597219 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0d4c8db-da49-4764-bde5-25686bbcf1e0 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:26:23.598644 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0d4c8db-da49-4764-bde5-25686bbcf1e0 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:26:23.616766 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 645/1281] 10.222.0.25 () {56 vars in 1143 bytes} [Sun Sep 25 04:26:23 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:26:23.623373 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcb9f266-431c-46ba-9926-608bb5e40926 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:26:23.711900 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:26:23.722111 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcb9f266-431c-46ba-9926-608bb5e40926 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:26:23.754509 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c267bf5-a9ae-453e-8694-d5fdd2f6d3e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:26:23.755106 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c267bf5-a9ae-453e-8694-d5fdd2f6d3e5 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:26:23.755390 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c267bf5-a9ae-453e-8694-d5fdd2f6d3e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:26:23.755664 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c267bf5-a9ae-453e-8694-d5fdd2f6d3e5 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:26:23.821587 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c267bf5-a9ae-453e-8694-d5fdd2f6d3e5 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:26:23.822965 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c267bf5-a9ae-453e-8694-d5fdd2f6d3e5 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:26:23.909990 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:26:23.921037 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 637/1282] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Sep 25 04:26:23 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 298 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 25 04:26:23.926855 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0d4c8db-da49-4764-bde5-25686bbcf1e0 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:26:23.987059 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:26:24.024701 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0d4c8db-da49-4764-bde5-25686bbcf1e0 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:26:24.057310 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b374da9-dfef-482b-970b-1aea2f7f9b04 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:26:24.058476 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b374da9-dfef-482b-970b-1aea2f7f9b04 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:26:24.058755 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b374da9-dfef-482b-970b-1aea2f7f9b04 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:26:24.059026 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b374da9-dfef-482b-970b-1aea2f7f9b04 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:26:24.068828 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b374da9-dfef-482b-970b-1aea2f7f9b04 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:26:24.070271 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b374da9-dfef-482b-970b-1aea2f7f9b04 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:26:24.085002 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-9b374da9-dfef-482b-970b-1aea2f7f9b04 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:26:24.085002 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:26:24.085002 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:26:24.085002 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:26:24.085002 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:26:24.085002 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:26:24.085002 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:26:24.085002 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:26:24.085002 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:26:24.085002 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:26:24.085002 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:26:24.085002 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:26:24.085002 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 25 04:26:24.085002 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 25 04:26:24.085002 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 25 04:26:24.085002 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 25 04:26:24.086157 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:26:24.086157 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:26:24.086157 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:26:24.086157 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:26:24.086157 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:26:24.086157 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:26:24.086157 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:26:24.086157 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:26:24.086157 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:26:24.086157 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:26:24.086157 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:26:24.086157 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:26:24.086157 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:26:24.086157 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:26:24.086157 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:26:24.086157 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:26:24.086157 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 25 04:26:24.087553 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 25 04:26:24.087553 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 25 04:26:24.087553 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:26:24.087553 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 646/1283] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:26:23 2022] GET /identity/v3/domains/Default => generated 87 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:26:24.091937 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c267bf5-a9ae-453e-8694-d5fdd2f6d3e5 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:26:24.189209 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c267bf5-a9ae-453e-8694-d5fdd2f6d3e5 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:26:24.222540 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85a0ecf4-625b-466a-a1aa-8f03541fe527 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:26:24.223139 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a0ecf4-625b-466a-a1aa-8f03541fe527 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:26:24.223418 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a0ecf4-625b-466a-a1aa-8f03541fe527 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:26:24.223689 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a0ecf4-625b-466a-a1aa-8f03541fe527 None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:26:24.224316 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85a0ecf4-625b-466a-a1aa-8f03541fe527 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:26:24.224619 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85a0ecf4-625b-466a-a1aa-8f03541fe527 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:26:24.226002 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85a0ecf4-625b-466a-a1aa-8f03541fe527 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:26:24.237441 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 638/1284] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Sep 25 04:26:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:26:24.243044 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b374da9-dfef-482b-970b-1aea2f7f9b04 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:26:24.338450 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b374da9-dfef-482b-970b-1aea2f7f9b04 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:26:24.351815 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:26:24.372102 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-898dde9e-4080-473b-a4c7-9dba1fcc05de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:26:24.372754 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-898dde9e-4080-473b-a4c7-9dba1fcc05de None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:26:24.373034 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-898dde9e-4080-473b-a4c7-9dba1fcc05de None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:26:24.373301 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-898dde9e-4080-473b-a4c7-9dba1fcc05de None admin] PATH_INFO: `/v3/projects/service` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:26:24.379660 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-898dde9e-4080-473b-a4c7-9dba1fcc05de None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:26:24.381129 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-898dde9e-4080-473b-a4c7-9dba1fcc05de None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:26:24.392153 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application [None req-898dde9e-4080-473b-a4c7-9dba1fcc05de None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 04:26:24.392153 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:26:24.392153 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:26:24.392153 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:26:24.392153 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:26:24.392153 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:26:24.392153 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:26:24.392153 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:26:24.392153 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:26:24.392153 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:26:24.392153 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:26:24.392153 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:26:24.392153 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 25 04:26:24.392153 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 25 04:26:24.392153 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 25 04:26:24.393245 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 25 04:26:24.393245 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:26:24.393245 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:26:24.393245 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:26:24.393245 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:26:24.393245 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:26:24.393245 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:26:24.393245 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:26:24.393245 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application with Lock( Sep 25 04:26:24.393245 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:26:24.393245 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:26:24.393245 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:26:24.393245 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:26:24.393245 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:26:24.393245 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:26:24.393245 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:26:24.393245 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:26:24.394372 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 25 04:26:24.394372 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 25 04:26:24.394372 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 25 04:26:24.394372 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 25 04:26:24.394372 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 25 04:26:24.394372 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 25 04:26:24.394372 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 25 04:26:24.394372 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: ERROR keystone.server.flask.application Sep 25 04:26:24.394372 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 647/1285] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Sep 25 04:26:24 2022] GET /identity/v3/projects/service => generated 88 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 25 04:26:24.399120 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85a0ecf4-625b-466a-a1aa-8f03541fe527 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:26:24.447444 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:26:24.497523 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85a0ecf4-625b-466a-a1aa-8f03541fe527 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:26:24.529759 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f9fb17e-ffd8-4480-bb12-cd539ccb25fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:26:24.530360 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f9fb17e-ffd8-4480-bb12-cd539ccb25fc None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:26:24.530640 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f9fb17e-ffd8-4480-bb12-cd539ccb25fc None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:26:24.530911 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f9fb17e-ffd8-4480-bb12-cd539ccb25fc None admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:26:24.531586 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f9fb17e-ffd8-4480-bb12-cd539ccb25fc None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:26:24.531891 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f9fb17e-ffd8-4480-bb12-cd539ccb25fc None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:26:24.533326 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f9fb17e-ffd8-4480-bb12-cd539ccb25fc None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:26:24.545608 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 639/1286] 10.222.0.25 () {56 vars in 1148 bytes} [Sun Sep 25 04:26:24 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:26:24.551599 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-898dde9e-4080-473b-a4c7-9dba1fcc05de None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:26:24.648894 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-898dde9e-4080-473b-a4c7-9dba1fcc05de None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:26:24.681222 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed7ceaea-5998-462c-8c72-ecc0fc53e87d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:26:24.681818 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed7ceaea-5998-462c-8c72-ecc0fc53e87d None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:26:24.682096 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed7ceaea-5998-462c-8c72-ecc0fc53e87d None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:26:24.682367 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed7ceaea-5998-462c-8c72-ecc0fc53e87d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:26:24.683738 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed7ceaea-5998-462c-8c72-ecc0fc53e87d None admin] RBAC: Adding query filter params (role.id=c51fe7c5ad804040879acfb54269455f, scope.project.id=056ca1b1a9974a7581550de0d9217823, user.id=676d04ef5a984edcaad63d9156568660) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:26:24.684045 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed7ceaea-5998-462c-8c72-ecc0fc53e87d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:26:24.685498 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed7ceaea-5998-462c-8c72-ecc0fc53e87d None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:26:24.691030 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 648/1287] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Sep 25 04:26:24 2022] GET /identity/v3/role_assignments?user.id=676d04ef5a984edcaad63d9156568660&scope.project.id=056ca1b1a9974a7581550de0d9217823&role.id=c51fe7c5ad804040879acfb54269455f => generated 608 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:26:26.854081 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88ce21e9-4caa-43b7-9f5f-6f8dcbf719a7 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:26:26.854647 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88ce21e9-4caa-43b7-9f5f-6f8dcbf719a7 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:26:26.854647 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88ce21e9-4caa-43b7-9f5f-6f8dcbf719a7 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:26:26.855883 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 640/1288] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:26:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:26:26.861166 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fbe8824-28f5-4f03-923f-30cd9426da04 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:26:26.861594 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fbe8824-28f5-4f03-923f-30cd9426da04 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:26:26.861752 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fbe8824-28f5-4f03-923f-30cd9426da04 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:26:26.880034 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-6fbe8824-28f5-4f03-923f-30cd9426da04 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:26:26.881258 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-6fbe8824-28f5-4f03-923f-30cd9426da04 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:26:26.911159 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:26:26.949003 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 649/1289] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:26:26 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 04:26:27.231202 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88ce21e9-4caa-43b7-9f5f-6f8dcbf719a7 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:26:27.329893 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88ce21e9-4caa-43b7-9f5f-6f8dcbf719a7 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:26:27.340842 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:26:27.364060 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32eba945-1135-462b-a0be-9e0c7799d8fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:26:27.364708 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32eba945-1135-462b-a0be-9e0c7799d8fe None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:26:27.364994 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32eba945-1135-462b-a0be-9e0c7799d8fe None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:26:27.365269 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32eba945-1135-462b-a0be-9e0c7799d8fe None admin] PATH_INFO: `/v3/services/placement` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:26:27.365844 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32eba945-1135-462b-a0be-9e0c7799d8fe None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:26:27.367208 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32eba945-1135-462b-a0be-9e0c7799d8fe None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:26:27.376071 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-32eba945-1135-462b-a0be-9e0c7799d8fe None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Sep 25 04:26:27.376071 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:26:27.376071 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 25 04:26:27.376071 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 25 04:26:27.376071 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 25 04:26:27.376071 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 25 04:26:27.376071 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 25 04:26:27.376071 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 04:26:27.376071 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 25 04:26:27.376071 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:26:27.376071 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 25 04:26:27.376071 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:26:27.376071 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:26:27.376071 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:26:27.376071 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:26:27.376071 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:26:27.376071 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:26:27.377319 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:26:27.377319 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:26:27.377319 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:26:27.377319 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:26:27.377319 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:26:27.377319 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:26:27.377319 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 25 04:26:27.377319 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 25 04:26:27.377319 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 25 04:26:27.377319 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 25 04:26:27.377319 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:26:27.377319 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:26:27.377319 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:26:27.377319 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:26:27.377319 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:26:27.377319 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:26:27.378215 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:26:27.378215 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:26:27.378215 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:26:27.378215 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:26:27.378215 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:26:27.378215 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:26:27.378215 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:26:27.378215 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:26:27.378215 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:26:27.378215 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:26:27.378215 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 25 04:26:27.378215 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 04:26:27.378215 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 25 04:26:27.378215 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:26:27.378215 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 641/1290] 10.222.0.25 () {56 vars in 1117 bytes} [Sun Sep 25 04:26:27 2022] GET /identity/v3/services/placement => generated 90 bytes in 146 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:26:27.383252 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fbe8824-28f5-4f03-923f-30cd9426da04 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:26:27.480359 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fbe8824-28f5-4f03-923f-30cd9426da04 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:26:27.513352 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7057ee4-37e8-4446-9962-27b1607efa51 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:26:27.513895 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7057ee4-37e8-4446-9962-27b1607efa51 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:26:27.514176 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7057ee4-37e8-4446-9962-27b1607efa51 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:26:27.514447 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7057ee4-37e8-4446-9962-27b1607efa51 None admin] PATH_INFO: `/v3/services` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:26:27.515071 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7057ee4-37e8-4446-9962-27b1607efa51 None admin] RBAC: Adding query filter params (name=placement) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:26:27.515374 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7057ee4-37e8-4446-9962-27b1607efa51 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:26:27.516759 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7057ee4-37e8-4446-9962-27b1607efa51 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:26:27.521425 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 650/1291] 10.222.0.25 () {56 vars in 1116 bytes} [Sun Sep 25 04:26:27 2022] GET /identity/v3/services?name=placement => generated 126 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:26:27.527740 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32eba945-1135-462b-a0be-9e0c7799d8fe None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:26:27.625586 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32eba945-1135-462b-a0be-9e0c7799d8fe None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:26:27.658980 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b65f771-1595-4b18-9d4b-bd2694427341 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:26:27.659575 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b65f771-1595-4b18-9d4b-bd2694427341 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:26:27.659858 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b65f771-1595-4b18-9d4b-bd2694427341 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:26:27.660130 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b65f771-1595-4b18-9d4b-bd2694427341 None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:26:27.660783 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b65f771-1595-4b18-9d4b-bd2694427341 None admin] RBAC: Adding query filter params (type=placement) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:26:27.661088 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b65f771-1595-4b18-9d4b-bd2694427341 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:26:27.662453 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b65f771-1595-4b18-9d4b-bd2694427341 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:26:27.666802 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 642/1292] 10.222.0.25 () {56 vars in 1116 bytes} [Sun Sep 25 04:26:27 2022] GET /identity/v3/services?type=placement => generated 126 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:26:29.810851 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f2626b3-fc47-4a85-9bf3-5771ae8196c3 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:26:29.811359 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f2626b3-fc47-4a85-9bf3-5771ae8196c3 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:26:29.811359 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f2626b3-fc47-4a85-9bf3-5771ae8196c3 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:26:29.812624 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 651/1293] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:26:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:26:29.817848 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4271afa7-8184-44e9-88ab-0e9609d6291a None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:26:29.818260 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4271afa7-8184-44e9-88ab-0e9609d6291a None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:26:29.818402 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4271afa7-8184-44e9-88ab-0e9609d6291a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:26:29.837893 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-4271afa7-8184-44e9-88ab-0e9609d6291a None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:26:29.839687 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-4271afa7-8184-44e9-88ab-0e9609d6291a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:26:29.868759 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:26:29.905979 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 643/1294] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:26:29 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 04:26:30.178178 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f2626b3-fc47-4a85-9bf3-5771ae8196c3 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:26:30.209076 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:26:30.276944 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f2626b3-fc47-4a85-9bf3-5771ae8196c3 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:26:30.309022 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a56bcad-8ea0-4765-908a-61065a554af8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:26:30.309612 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a56bcad-8ea0-4765-908a-61065a554af8 None admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:26:30.309886 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a56bcad-8ea0-4765-908a-61065a554af8 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:26:30.310150 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a56bcad-8ea0-4765-908a-61065a554af8 None admin] PATH_INFO: `/v3/services` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:26:30.310795 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a56bcad-8ea0-4765-908a-61065a554af8 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:26:30.312134 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a56bcad-8ea0-4765-908a-61065a554af8 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:26:30.325879 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 652/1295] 10.222.0.25 () {60 vars in 1141 bytes} [Sun Sep 25 04:26:30 2022] POST /identity/v3/services => generated 246 bytes in 148 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:26:32.558020 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b4788de-472c-4375-9fd0-6569a6ee573e None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:26:32.558657 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b4788de-472c-4375-9fd0-6569a6ee573e None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:26:32.558657 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b4788de-472c-4375-9fd0-6569a6ee573e None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:26:32.560233 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 644/1296] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:26:32 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:26:32.565684 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d153a99a-8d54-40af-93d3-cd1cfa81f24e None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:26:32.566134 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d153a99a-8d54-40af-93d3-cd1cfa81f24e None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:26:32.566248 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d153a99a-8d54-40af-93d3-cd1cfa81f24e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:26:32.585282 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-d153a99a-8d54-40af-93d3-cd1cfa81f24e None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:26:32.586527 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-d153a99a-8d54-40af-93d3-cd1cfa81f24e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:26:32.674892 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 653/1297] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:26:32 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 25 04:26:32.961856 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b4788de-472c-4375-9fd0-6569a6ee573e None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:26:33.061437 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b4788de-472c-4375-9fd0-6569a6ee573e None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:26:33.094796 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-254a716c-d701-48e9-99c7-cb909a956f85 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:26:33.095421 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-254a716c-d701-48e9-99c7-cb909a956f85 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:26:33.095703 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-254a716c-d701-48e9-99c7-cb909a956f85 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:26:33.095976 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-254a716c-d701-48e9-99c7-cb909a956f85 None admin] PATH_INFO: `/v3/services/placement` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:26:33.096552 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-254a716c-d701-48e9-99c7-cb909a956f85 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:26:33.097946 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-254a716c-d701-48e9-99c7-cb909a956f85 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:26:33.107171 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-254a716c-d701-48e9-99c7-cb909a956f85 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Sep 25 04:26:33.107171 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:26:33.107171 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 25 04:26:33.107171 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 25 04:26:33.107171 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 25 04:26:33.107171 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 25 04:26:33.107171 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 25 04:26:33.107171 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 04:26:33.107171 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 25 04:26:33.107171 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:26:33.107171 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 25 04:26:33.107171 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:26:33.107171 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:26:33.107171 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:26:33.107171 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:26:33.107171 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:26:33.107171 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:26:33.108457 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:26:33.108457 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:26:33.108457 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:26:33.108457 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:26:33.108457 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:26:33.108457 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:26:33.108457 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 25 04:26:33.108457 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 25 04:26:33.108457 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 25 04:26:33.108457 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 25 04:26:33.108457 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:26:33.108457 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:26:33.108457 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:26:33.108457 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:26:33.108457 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:26:33.108457 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:26:33.109711 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:26:33.109711 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:26:33.109711 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:26:33.109711 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:26:33.109711 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:26:33.109711 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:26:33.109711 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:26:33.109711 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:26:33.109711 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:26:33.109711 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:26:33.109711 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 25 04:26:33.109711 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 04:26:33.109711 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 25 04:26:33.109711 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:26:33.109711 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 645/1298] 10.222.0.25 () {56 vars in 1117 bytes} [Sun Sep 25 04:26:32 2022] GET /identity/v3/services/placement => generated 90 bytes in 147 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:26:33.115180 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d153a99a-8d54-40af-93d3-cd1cfa81f24e None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:26:33.135481 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:26:33.231029 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d153a99a-8d54-40af-93d3-cd1cfa81f24e None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:26:33.264761 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0773db6-f46b-4dc3-a176-17c275639c82 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:26:33.265373 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0773db6-f46b-4dc3-a176-17c275639c82 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:26:33.265655 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0773db6-f46b-4dc3-a176-17c275639c82 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:26:33.265928 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0773db6-f46b-4dc3-a176-17c275639c82 None admin] PATH_INFO: `/v3/services` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:26:33.266554 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0773db6-f46b-4dc3-a176-17c275639c82 None admin] RBAC: Adding query filter params (name=placement) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:26:33.266860 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0773db6-f46b-4dc3-a176-17c275639c82 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:26:33.268219 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0773db6-f46b-4dc3-a176-17c275639c82 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:26:33.273389 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 654/1299] 10.222.0.25 () {56 vars in 1116 bytes} [Sun Sep 25 04:26:33 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:26:33.279553 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-254a716c-d701-48e9-99c7-cb909a956f85 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:26:33.310798 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:26:33.377599 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-254a716c-d701-48e9-99c7-cb909a956f85 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:26:33.409913 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-020d7d6b-5f1d-4619-aec5-3d9a4bedbf11 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:26:33.410509 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-020d7d6b-5f1d-4619-aec5-3d9a4bedbf11 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:26:33.410789 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-020d7d6b-5f1d-4619-aec5-3d9a4bedbf11 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:26:33.411060 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-020d7d6b-5f1d-4619-aec5-3d9a4bedbf11 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:26:33.411745 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-020d7d6b-5f1d-4619-aec5-3d9a4bedbf11 None admin] RBAC: Adding query filter params (interface=public, service_id=23222b0969f04841855e6cf32a9c88ad, region_id=RegionOne) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:26:33.412053 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-020d7d6b-5f1d-4619-aec5-3d9a4bedbf11 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:26:33.413432 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-020d7d6b-5f1d-4619-aec5-3d9a4bedbf11 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:26:33.418297 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 646/1300] 10.222.0.25 () {56 vars in 1251 bytes} [Sun Sep 25 04:26:33 2022] GET /identity/v3/endpoints?service_id=23222b0969f04841855e6cf32a9c88ad&interface=public®ion_id=RegionOne => generated 194 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:26:33.423912 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0773db6-f46b-4dc3-a176-17c275639c82 None admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:26:33.506110 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:26:33.524658 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0773db6-f46b-4dc3-a176-17c275639c82 None admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:26:33.557800 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84a3d3a0-fc7b-48f8-b992-444f9f924327 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:26:33.558393 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84a3d3a0-fc7b-48f8-b992-444f9f924327 None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:26:33.558678 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84a3d3a0-fc7b-48f8-b992-444f9f924327 None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:26:33.558949 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84a3d3a0-fc7b-48f8-b992-444f9f924327 None admin] PATH_INFO: `/v3/services` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:26:33.559544 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84a3d3a0-fc7b-48f8-b992-444f9f924327 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:26:33.560930 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84a3d3a0-fc7b-48f8-b992-444f9f924327 None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:26:33.567440 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 655/1301] 10.222.0.25 () {56 vars in 1087 bytes} [Sun Sep 25 04:26:33 2022] GET /identity/v3/services => generated 2191 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:26:35.710742 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7922098e-e397-4dcc-892a-485871a11d2b None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:26:35.711301 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7922098e-e397-4dcc-892a-485871a11d2b None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:26:35.711301 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7922098e-e397-4dcc-892a-485871a11d2b None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:26:35.712661 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 647/1302] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:26:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:26:35.718564 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97a8ebd5-a101-4e18-886e-c7e68d25dc2d None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:26:35.718986 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97a8ebd5-a101-4e18-886e-c7e68d25dc2d None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:26:35.719106 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97a8ebd5-a101-4e18-886e-c7e68d25dc2d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:26:35.737722 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-97a8ebd5-a101-4e18-886e-c7e68d25dc2d None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:26:35.738825 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-97a8ebd5-a101-4e18-886e-c7e68d25dc2d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:26:35.803507 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 656/1303] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:26:35 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 25 04:26:36.075742 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7922098e-e397-4dcc-892a-485871a11d2b None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:26:36.168877 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:26:36.175906 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7922098e-e397-4dcc-892a-485871a11d2b None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:26:36.209442 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d350893-73ad-4d6a-b551-0eb82d6bd4c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:26:36.210059 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d350893-73ad-4d6a-b551-0eb82d6bd4c3 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:26:36.210339 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d350893-73ad-4d6a-b551-0eb82d6bd4c3 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:26:36.210616 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d350893-73ad-4d6a-b551-0eb82d6bd4c3 None admin] PATH_INFO: `/v3/services/placement` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:26:36.211188 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d350893-73ad-4d6a-b551-0eb82d6bd4c3 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:26:36.212540 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d350893-73ad-4d6a-b551-0eb82d6bd4c3 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:26:36.221075 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-8d350893-73ad-4d6a-b551-0eb82d6bd4c3 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Sep 25 04:26:36.221075 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:26:36.221075 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 25 04:26:36.221075 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 25 04:26:36.221075 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 25 04:26:36.221075 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 25 04:26:36.221075 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 25 04:26:36.221075 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 04:26:36.221075 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 25 04:26:36.221075 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:26:36.221075 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 25 04:26:36.221075 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:26:36.221075 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 25 04:26:36.221075 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 25 04:26:36.221075 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 25 04:26:36.221075 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 25 04:26:36.221075 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 25 04:26:36.222265 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 25 04:26:36.222265 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 25 04:26:36.222265 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 25 04:26:36.222265 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 25 04:26:36.222265 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 25 04:26:36.222265 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 25 04:26:36.222265 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 25 04:26:36.222265 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 25 04:26:36.222265 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 25 04:26:36.222265 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 25 04:26:36.222265 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 25 04:26:36.222265 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 25 04:26:36.222265 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 25 04:26:36.222265 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 25 04:26:36.222265 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 25 04:26:36.222265 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Sep 25 04:26:36.223156 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 25 04:26:36.223156 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Sep 25 04:26:36.223156 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 25 04:26:36.223156 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Sep 25 04:26:36.223156 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 25 04:26:36.223156 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 25 04:26:36.223156 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 25 04:26:36.223156 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Sep 25 04:26:36.223156 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 25 04:26:36.223156 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Sep 25 04:26:36.223156 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 25 04:26:36.223156 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 25 04:26:36.223156 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 25 04:26:36.223156 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Sep 25 04:26:36.223156 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 648/1304] 10.222.0.25 () {56 vars in 1117 bytes} [Sun Sep 25 04:26:36 2022] GET /identity/v3/services/placement => generated 90 bytes in 147 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 25 04:26:36.228086 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97a8ebd5-a101-4e18-886e-c7e68d25dc2d None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:26:36.302563 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:26:36.325972 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97a8ebd5-a101-4e18-886e-c7e68d25dc2d None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:26:36.358204 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f42b0743-b7f0-4c3e-89e2-74cc7afaf54f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:26:36.358793 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f42b0743-b7f0-4c3e-89e2-74cc7afaf54f None admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:26:36.359071 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f42b0743-b7f0-4c3e-89e2-74cc7afaf54f None admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:26:36.359338 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f42b0743-b7f0-4c3e-89e2-74cc7afaf54f None admin] PATH_INFO: `/v3/services` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:26:36.359952 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f42b0743-b7f0-4c3e-89e2-74cc7afaf54f None admin] RBAC: Adding query filter params (name=placement) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:26:36.360250 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f42b0743-b7f0-4c3e-89e2-74cc7afaf54f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:26:36.361628 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f42b0743-b7f0-4c3e-89e2-74cc7afaf54f None admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:26:36.366541 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 657/1305] 10.222.0.25 () {56 vars in 1116 bytes} [Sun Sep 25 04:26:36 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:26:36.372831 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d350893-73ad-4d6a-b551-0eb82d6bd4c3 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:26:36.472430 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d350893-73ad-4d6a-b551-0eb82d6bd4c3 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:26:36.506214 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-070dd759-b195-4188-b80c-3104529189c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:26:36.506818 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-070dd759-b195-4188-b80c-3104529189c8 None admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:26:36.507100 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-070dd759-b195-4188-b80c-3104529189c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:26:36.507379 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-070dd759-b195-4188-b80c-3104529189c8 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:26:36.508042 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-070dd759-b195-4188-b80c-3104529189c8 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:26:36.509444 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-070dd759-b195-4188-b80c-3104529189c8 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:26:36.529133 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 649/1306] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:26:36 2022] POST /identity/v3/endpoints => generated 331 bytes in 157 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:26:46.363615 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=30 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:26:46.524884 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=91,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:27:40.866925 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e35adca-b646-463e-b557-49b2a7543433 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:27:40.867915 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e35adca-b646-463e-b557-49b2a7543433 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:27:40.869200 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e35adca-b646-463e-b557-49b2a7543433 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:27:40.872910 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 658/1307] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:27:40 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:27:40.876986 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0151d491-bbf2-415d-976a-74c3935a1515 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:27:40.877430 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0151d491-bbf2-415d-976a-74c3935a1515 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:27:40.877539 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0151d491-bbf2-415d-976a-74c3935a1515 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:27:40.953505 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-0151d491-bbf2-415d-976a-74c3935a1515 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:27:40.954990 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-0151d491-bbf2-415d-976a-74c3935a1515 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:27:41.083303 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 650/1308] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:27:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 207 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:27:41.380387 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a50b2e7-df97-4d0e-ac2a-7f44edf8ed2f None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:27:41.380870 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a50b2e7-df97-4d0e-ac2a-7f44edf8ed2f None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:27:41.381221 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a50b2e7-df97-4d0e-ac2a-7f44edf8ed2f None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:27:41.382681 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 659/1309] 10.222.0.25 () {54 vars in 964 bytes} [Sun Sep 25 04:27:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:27:41.389333 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef9c772a-4ef4-4768-90ea-e9e5d74b8c60 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:27:41.389697 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef9c772a-4ef4-4768-90ea-e9e5d74b8c60 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:27:41.390046 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef9c772a-4ef4-4768-90ea-e9e5d74b8c60 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:27:41.471421 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-ef9c772a-4ef4-4768-90ea-e9e5d74b8c60 None None] MFA Rules not processed for user `0268e88e0c5645a98bab5664be6e49bc`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:27:41.472768 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-ef9c772a-4ef4-4768-90ea-e9e5d74b8c60 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:27:41.496898 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:27:41.556832 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 651/1310] 10.222.0.25 () {58 vars in 1061 bytes} [Sun Sep 25 04:27:41 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:27:41.566460 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a50b2e7-df97-4d0e-ac2a-7f44edf8ed2f None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:27:41.568562 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-5a50b2e7-df97-4d0e-ac2a-7f44edf8ed2f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:27:41.705296 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a50b2e7-df97-4d0e-ac2a-7f44edf8ed2f None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:27:41.796624 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-183c9239-6fa4-485c-9736-65130815a48f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:27:41.797368 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-183c9239-6fa4-485c-9736-65130815a48f service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:27:41.797650 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-183c9239-6fa4-485c-9736-65130815a48f service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:27:41.797922 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-183c9239-6fa4-485c-9736-65130815a48f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:27:41.799570 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-183c9239-6fa4-485c-9736-65130815a48f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:27:41.859070 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:27:41.918360 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-183c9239-6fa4-485c-9736-65130815a48f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:27:41.919809 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-183c9239-6fa4-485c-9736-65130815a48f service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:27:41.939448 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 660/1311] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:27:41 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 373 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:27:44.414840 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b34d66-37a7-40bf-851a-5ae647fe7434 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:27:44.415345 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b34d66-37a7-40bf-851a-5ae647fe7434 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:27:44.415675 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b34d66-37a7-40bf-851a-5ae647fe7434 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:27:44.417023 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 652/1312] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:27:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:27:44.422540 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aa741aa-2743-4bb1-b9a4-6756137b2714 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:27:44.422960 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aa741aa-2743-4bb1-b9a4-6756137b2714 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:27:44.423088 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aa741aa-2743-4bb1-b9a4-6756137b2714 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:27:44.509342 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-7aa741aa-2743-4bb1-b9a4-6756137b2714 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:27:44.510555 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-7aa741aa-2743-4bb1-b9a4-6756137b2714 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:27:44.606980 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 661/1313] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:27:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:27:44.891860 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6b34d66-37a7-40bf-851a-5ae647fe7434 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:27:44.893561 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-e6b34d66-37a7-40bf-851a-5ae647fe7434 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:27:45.000316 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6b34d66-37a7-40bf-851a-5ae647fe7434 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:27:45.073668 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:27:45.076005 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24efe336-ffe5-4b42-98b6-ad22ba631bf4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:27:45.076826 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24efe336-ffe5-4b42-98b6-ad22ba631bf4 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:27:45.077280 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24efe336-ffe5-4b42-98b6-ad22ba631bf4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:27:45.077749 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24efe336-ffe5-4b42-98b6-ad22ba631bf4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:27:45.079349 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-24efe336-ffe5-4b42-98b6-ad22ba631bf4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:27:45.187146 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24efe336-ffe5-4b42-98b6-ad22ba631bf4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:27:45.188575 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24efe336-ffe5-4b42-98b6-ad22ba631bf4 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:27:45.207657 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 653/1314] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:27:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 316 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:27:47.664562 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0190685-2c02-49d3-ad3b-f1bd5a2654a7 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:27:47.665114 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0190685-2c02-49d3-ad3b-f1bd5a2654a7 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:27:47.665216 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0190685-2c02-49d3-ad3b-f1bd5a2654a7 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:27:47.666597 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 662/1315] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:27:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:27:47.671939 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d3d9946-4881-40bc-9e21-1e3902dcb487 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:27:47.672381 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d3d9946-4881-40bc-9e21-1e3902dcb487 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:27:47.672478 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d3d9946-4881-40bc-9e21-1e3902dcb487 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:27:47.737424 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-1d3d9946-4881-40bc-9e21-1e3902dcb487 None None] MFA Rules not processed for user `bc82e617c36e4fdfa13e159395d54e1c`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:27:47.738551 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-1d3d9946-4881-40bc-9e21-1e3902dcb487 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:27:47.789462 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:27:47.834525 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 654/1316] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:27:47 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:27:48.122325 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0190685-2c02-49d3-ad3b-f1bd5a2654a7 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:27:48.123918 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-b0190685-2c02-49d3-ad3b-f1bd5a2654a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:27:48.133613 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:27:48.227052 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0190685-2c02-49d3-ad3b-f1bd5a2654a7 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:27:48.299429 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43995bf6-b30f-413f-91d6-30b0468e2399 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:27:48.300032 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43995bf6-b30f-413f-91d6-30b0468e2399 service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:27:48.300312 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43995bf6-b30f-413f-91d6-30b0468e2399 service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:27:48.300584 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43995bf6-b30f-413f-91d6-30b0468e2399 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:27:48.302215 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-43995bf6-b30f-413f-91d6-30b0468e2399 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:27:48.364093 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:27:48.410438 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43995bf6-b30f-413f-91d6-30b0468e2399 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:27:48.411873 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43995bf6-b30f-413f-91d6-30b0468e2399 service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:27:48.431911 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 663/1317] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:27:48 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:27:51.081034 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18cf085d-f48e-4f1d-94c0-a84e436bd30c None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:27:51.081576 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18cf085d-f48e-4f1d-94c0-a84e436bd30c None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:27:51.081576 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18cf085d-f48e-4f1d-94c0-a84e436bd30c None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:27:51.082960 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 655/1318] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:27:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:27:51.089067 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd936253-eb1c-4b2f-b948-28266677abc5 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:27:51.089455 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd936253-eb1c-4b2f-b948-28266677abc5 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:27:51.089619 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd936253-eb1c-4b2f-b948-28266677abc5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:27:51.155177 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-fd936253-eb1c-4b2f-b948-28266677abc5 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:27:51.156378 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-fd936253-eb1c-4b2f-b948-28266677abc5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:27:51.251187 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 664/1319] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:27:51 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:27:51.537015 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18cf085d-f48e-4f1d-94c0-a84e436bd30c None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:27:51.538571 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-18cf085d-f48e-4f1d-94c0-a84e436bd30c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:27:51.644477 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18cf085d-f48e-4f1d-94c0-a84e436bd30c None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:27:51.720913 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ffa85d7-2581-4c11-95f7-8ffe3e97b248 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:27:51.721521 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ffa85d7-2581-4c11-95f7-8ffe3e97b248 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:27:51.721801 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ffa85d7-2581-4c11-95f7-8ffe3e97b248 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:27:51.722074 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ffa85d7-2581-4c11-95f7-8ffe3e97b248 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:27:51.723655 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-7ffa85d7-2581-4c11-95f7-8ffe3e97b248 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:27:51.731154 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:27:51.832406 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ffa85d7-2581-4c11-95f7-8ffe3e97b248 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:27:51.833879 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ffa85d7-2581-4c11-95f7-8ffe3e97b248 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:27:51.851482 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 656/1320] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:27:51 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 315 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:27:54.644703 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae39df3-4305-4d6b-a7d2-fb00c1eba875 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:27:54.645223 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae39df3-4305-4d6b-a7d2-fb00c1eba875 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:27:54.645334 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae39df3-4305-4d6b-a7d2-fb00c1eba875 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:27:54.646593 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 665/1321] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:27:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:27:54.651952 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c033fc7d-cb5b-473d-b10e-2013ba482981 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:27:54.652399 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c033fc7d-cb5b-473d-b10e-2013ba482981 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:27:54.652508 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c033fc7d-cb5b-473d-b10e-2013ba482981 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:27:54.724115 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-c033fc7d-cb5b-473d-b10e-2013ba482981 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:27:54.725394 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-c033fc7d-cb5b-473d-b10e-2013ba482981 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:27:54.787284 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:27:54.830755 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 657/1322] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:27:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:27:55.111949 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fae39df3-4305-4d6b-a7d2-fb00c1eba875 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:27:55.113532 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-fae39df3-4305-4d6b-a7d2-fb00c1eba875 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:27:55.133779 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:27:55.217177 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fae39df3-4305-4d6b-a7d2-fb00c1eba875 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:27:55.291656 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c874233-4b0c-4f8d-a3c1-5f8766f9f9b0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:27:55.292259 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c874233-4b0c-4f8d-a3c1-5f8766f9f9b0 service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:27:55.292534 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c874233-4b0c-4f8d-a3c1-5f8766f9f9b0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:27:55.292841 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c874233-4b0c-4f8d-a3c1-5f8766f9f9b0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:27:55.294418 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-8c874233-4b0c-4f8d-a3c1-5f8766f9f9b0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:27:55.365787 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:27:55.404919 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c874233-4b0c-4f8d-a3c1-5f8766f9f9b0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:27:55.406352 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c874233-4b0c-4f8d-a3c1-5f8766f9f9b0 service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:27:55.424340 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 666/1323] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:27:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 313 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:27:57.643507 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-001d3a8d-1b7c-4784-a35e-988cf6b63b9d None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:27:57.644068 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-001d3a8d-1b7c-4784-a35e-988cf6b63b9d None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:27:57.644068 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-001d3a8d-1b7c-4784-a35e-988cf6b63b9d None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:27:57.645505 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 658/1324] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:27:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:27:57.650835 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14a13bce-b2de-4370-aa12-f1344fd3727b None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:27:57.651315 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14a13bce-b2de-4370-aa12-f1344fd3727b None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:27:57.651416 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14a13bce-b2de-4370-aa12-f1344fd3727b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:27:57.718429 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-14a13bce-b2de-4370-aa12-f1344fd3727b None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:27:57.719628 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-14a13bce-b2de-4370-aa12-f1344fd3727b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:27:57.824460 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 667/1325] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:27:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:27:58.118905 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8fbe1af-3c0c-4f1f-9d3a-82c93df70a30 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:27:58.119635 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8fbe1af-3c0c-4f1f-9d3a-82c93df70a30 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:27:58.119918 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8fbe1af-3c0c-4f1f-9d3a-82c93df70a30 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:27:58.121773 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 659/1326] 10.222.0.25 () {54 vars in 964 bytes} [Sun Sep 25 04:27:58 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:27:58.127753 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af0683c7-c59d-4f0a-b30c-0d579aa40078 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:27:58.128115 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af0683c7-c59d-4f0a-b30c-0d579aa40078 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:27:58.128456 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af0683c7-c59d-4f0a-b30c-0d579aa40078 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:27:58.160167 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:27:58.207522 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-af0683c7-c59d-4f0a-b30c-0d579aa40078 None None] MFA Rules not processed for user `0268e88e0c5645a98bab5664be6e49bc`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:27:58.208748 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-af0683c7-c59d-4f0a-b30c-0d579aa40078 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:27:58.284421 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 668/1327] 10.222.0.25 () {58 vars in 1061 bytes} [Sun Sep 25 04:27:58 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 04:27:58.294245 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8fbe1af-3c0c-4f1f-9d3a-82c93df70a30 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:27:58.295805 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-b8fbe1af-3c0c-4f1f-9d3a-82c93df70a30 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:27:58.400085 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8fbe1af-3c0c-4f1f-9d3a-82c93df70a30 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:27:58.478352 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46e5a0fe-e2c2-419b-a5aa-eb93c4d2ca1c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:27:58.478352 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46e5a0fe-e2c2-419b-a5aa-eb93c4d2ca1c service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:27:58.478352 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46e5a0fe-e2c2-419b-a5aa-eb93c4d2ca1c service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:27:58.478352 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46e5a0fe-e2c2-419b-a5aa-eb93c4d2ca1c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:27:58.480292 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-46e5a0fe-e2c2-419b-a5aa-eb93c4d2ca1c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:27:58.498306 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:27:58.594562 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46e5a0fe-e2c2-419b-a5aa-eb93c4d2ca1c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:27:58.595980 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46e5a0fe-e2c2-419b-a5aa-eb93c4d2ca1c service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:27:58.614004 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 660/1328] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:27:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 320 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:28:01.077031 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb97a7e2-2ab0-4bbd-b64b-466b5a493929 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:28:01.077578 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb97a7e2-2ab0-4bbd-b64b-466b5a493929 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:28:01.077723 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb97a7e2-2ab0-4bbd-b64b-466b5a493929 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:28:01.079152 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 669/1329] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:28:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:28:01.084976 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b5b4c10-fe80-48fc-8ab1-0d5bbae70ff9 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:28:01.085483 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b5b4c10-fe80-48fc-8ab1-0d5bbae70ff9 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:28:01.085575 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b5b4c10-fe80-48fc-8ab1-0d5bbae70ff9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:28:01.157399 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-3b5b4c10-fe80-48fc-8ab1-0d5bbae70ff9 None None] MFA Rules not processed for user `bc82e617c36e4fdfa13e159395d54e1c`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:28:01.158791 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-3b5b4c10-fe80-48fc-8ab1-0d5bbae70ff9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:28:01.229232 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:28:01.260500 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 661/1330] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:28:01 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:28:01.547824 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb97a7e2-2ab0-4bbd-b64b-466b5a493929 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:28:01.549466 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-bb97a7e2-2ab0-4bbd-b64b-466b5a493929 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:28:01.653031 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb97a7e2-2ab0-4bbd-b64b-466b5a493929 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:28:01.674798 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:28:01.735282 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7708136-1bb2-42c0-8bdb-7efd2ebe5cec service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:28:01.735808 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7708136-1bb2-42c0-8bdb-7efd2ebe5cec service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:28:01.736086 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7708136-1bb2-42c0-8bdb-7efd2ebe5cec service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:28:01.736352 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7708136-1bb2-42c0-8bdb-7efd2ebe5cec service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:28:01.737966 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-e7708136-1bb2-42c0-8bdb-7efd2ebe5cec service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:28:01.854013 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7708136-1bb2-42c0-8bdb-7efd2ebe5cec service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:28:01.855519 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7708136-1bb2-42c0-8bdb-7efd2ebe5cec service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:28:01.875152 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 670/1331] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:28:01 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 328 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:28:04.441502 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba9fb910-1337-4f17-bd20-c27f1e1ffc7d None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:28:04.441990 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba9fb910-1337-4f17-bd20-c27f1e1ffc7d None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:28:04.442084 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba9fb910-1337-4f17-bd20-c27f1e1ffc7d None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:28:04.443384 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 662/1332] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:28:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:28:04.449259 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f896c51-291f-4755-ad10-25a17e66811f None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:28:04.449757 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f896c51-291f-4755-ad10-25a17e66811f None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:28:04.449857 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f896c51-291f-4755-ad10-25a17e66811f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:28:04.513362 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:28:04.516241 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-6f896c51-291f-4755-ad10-25a17e66811f None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:28:04.518069 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-6f896c51-291f-4755-ad10-25a17e66811f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:28:04.612604 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 671/1333] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:28:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:28:04.897551 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba9fb910-1337-4f17-bd20-c27f1e1ffc7d None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:28:04.899095 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-ba9fb910-1337-4f17-bd20-c27f1e1ffc7d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:28:05.001298 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba9fb910-1337-4f17-bd20-c27f1e1ffc7d None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:28:05.073911 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29cf012c-2691-4ca2-9bc0-43a86c07936e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:28:05.074518 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29cf012c-2691-4ca2-9bc0-43a86c07936e service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:28:05.074800 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29cf012c-2691-4ca2-9bc0-43a86c07936e service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:28:05.075073 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29cf012c-2691-4ca2-9bc0-43a86c07936e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:28:05.076685 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-29cf012c-2691-4ca2-9bc0-43a86c07936e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:28:05.103741 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:28:05.187398 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29cf012c-2691-4ca2-9bc0-43a86c07936e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:28:05.188786 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29cf012c-2691-4ca2-9bc0-43a86c07936e service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:28:05.207476 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 663/1334] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:28:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:28:07.400127 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9389447-e073-496e-bec8-e7b94c84be6c None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:28:07.400726 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9389447-e073-496e-bec8-e7b94c84be6c None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:28:07.400726 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9389447-e073-496e-bec8-e7b94c84be6c None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:28:07.402024 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 672/1335] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:28:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:28:07.407314 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59f9f96c-d0f5-4f79-b2ba-fb79d9446ca6 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:28:07.407759 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59f9f96c-d0f5-4f79-b2ba-fb79d9446ca6 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:28:07.407856 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59f9f96c-d0f5-4f79-b2ba-fb79d9446ca6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:28:07.473701 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-59f9f96c-d0f5-4f79-b2ba-fb79d9446ca6 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:28:07.474941 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-59f9f96c-d0f5-4f79-b2ba-fb79d9446ca6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:28:07.549469 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:28:07.574630 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 664/1336] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:28:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:28:07.861352 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9389447-e073-496e-bec8-e7b94c84be6c None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:28:07.862919 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-d9389447-e073-496e-bec8-e7b94c84be6c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:28:07.964717 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9389447-e073-496e-bec8-e7b94c84be6c None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:28:07.992465 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:28:08.037782 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af1adfa1-b433-4100-a7ee-b71169911895 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:28:08.038380 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af1adfa1-b433-4100-a7ee-b71169911895 service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:28:08.038654 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af1adfa1-b433-4100-a7ee-b71169911895 service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:28:08.038920 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af1adfa1-b433-4100-a7ee-b71169911895 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:28:08.041404 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-af1adfa1-b433-4100-a7ee-b71169911895 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:28:08.149004 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af1adfa1-b433-4100-a7ee-b71169911895 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:28:08.150747 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af1adfa1-b433-4100-a7ee-b71169911895 service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:28:08.168927 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 673/1337] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:28:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 308 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:28:10.651681 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac4c7f7e-269c-49ea-8bcd-9f73afe55591 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:28:10.652270 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac4c7f7e-269c-49ea-8bcd-9f73afe55591 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:28:10.652270 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac4c7f7e-269c-49ea-8bcd-9f73afe55591 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:28:10.653551 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 665/1338] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:28:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:28:10.658664 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff5de578-a0e3-4b90-b24d-26a7ccabcebf None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:28:10.659134 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff5de578-a0e3-4b90-b24d-26a7ccabcebf None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:28:10.659242 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff5de578-a0e3-4b90-b24d-26a7ccabcebf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:28:10.724817 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-ff5de578-a0e3-4b90-b24d-26a7ccabcebf None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:28:10.726017 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-ff5de578-a0e3-4b90-b24d-26a7ccabcebf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:28:10.734752 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:28:10.821961 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 674/1339] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:28:10 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:28:11.106817 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac4c7f7e-269c-49ea-8bcd-9f73afe55591 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:28:11.108363 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-ac4c7f7e-269c-49ea-8bcd-9f73afe55591 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:28:11.210701 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac4c7f7e-269c-49ea-8bcd-9f73afe55591 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:28:11.282011 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb59179f-a60f-46bd-824e-58b29a4fd219 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:28:11.282548 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb59179f-a60f-46bd-824e-58b29a4fd219 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:28:11.282828 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb59179f-a60f-46bd-824e-58b29a4fd219 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:28:11.283101 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb59179f-a60f-46bd-824e-58b29a4fd219 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:28:11.284712 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-bb59179f-a60f-46bd-824e-58b29a4fd219 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:28:11.319026 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:28:11.392208 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb59179f-a60f-46bd-824e-58b29a4fd219 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:28:11.393657 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb59179f-a60f-46bd-824e-58b29a4fd219 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:28:11.411930 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 666/1340] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:28:11 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 306 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 25 04:28:15.542833 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e61544c-4e41-4549-8889-701bc2497d1a None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:28:15.543394 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e61544c-4e41-4549-8889-701bc2497d1a None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:28:15.543653 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e61544c-4e41-4549-8889-701bc2497d1a None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:28:15.545412 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 675/1341] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:28:15 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:28:15.551991 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41a0ed95-930c-471c-b847-c3fc5296fefe None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:28:15.552899 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41a0ed95-930c-471c-b847-c3fc5296fefe None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:28:15.553519 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41a0ed95-930c-471c-b847-c3fc5296fefe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:28:15.624488 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-41a0ed95-930c-471c-b847-c3fc5296fefe None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:28:15.625782 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-41a0ed95-930c-471c-b847-c3fc5296fefe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:28:15.724573 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:28:15.736508 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 667/1342] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:28:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:28:16.032287 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e61544c-4e41-4549-8889-701bc2497d1a None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:28:16.033972 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-0e61544c-4e41-4549-8889-701bc2497d1a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:28:16.141700 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e61544c-4e41-4549-8889-701bc2497d1a None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:28:16.188591 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:28:16.225927 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9736ec2-a140-4a9b-9a49-973505b48945 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:28:16.226534 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9736ec2-a140-4a9b-9a49-973505b48945 service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:28:16.226809 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9736ec2-a140-4a9b-9a49-973505b48945 service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:28:16.227075 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9736ec2-a140-4a9b-9a49-973505b48945 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:28:16.228910 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-c9736ec2-a140-4a9b-9a49-973505b48945 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:28:16.341433 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9736ec2-a140-4a9b-9a49-973505b48945 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:28:16.342784 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9736ec2-a140-4a9b-9a49-973505b48945 service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:28:16.361286 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 676/1343] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:28:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 330 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:28:19.214990 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a33fa264-e6a5-40a3-ab63-04f907e041dc None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:28:19.215560 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a33fa264-e6a5-40a3-ab63-04f907e041dc None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:28:19.215560 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a33fa264-e6a5-40a3-ab63-04f907e041dc None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:28:19.216915 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 668/1344] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:28:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:28:19.222191 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22a4cacd-62f8-4864-a194-32d327f89482 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:28:19.222618 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22a4cacd-62f8-4864-a194-32d327f89482 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:28:19.222880 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22a4cacd-62f8-4864-a194-32d327f89482 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:28:19.290190 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-22a4cacd-62f8-4864-a194-32d327f89482 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:28:19.291418 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-22a4cacd-62f8-4864-a194-32d327f89482 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:28:19.309809 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:28:19.393169 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 677/1345] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:28:19 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:28:19.678452 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a33fa264-e6a5-40a3-ab63-04f907e041dc None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:28:19.680000 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-a33fa264-e6a5-40a3-ab63-04f907e041dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:28:19.783660 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a33fa264-e6a5-40a3-ab63-04f907e041dc None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:28:19.858508 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5b23b40-0cf4-4b1f-afaf-7bba7505c10c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:28:19.859133 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5b23b40-0cf4-4b1f-afaf-7bba7505c10c service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:28:19.859422 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5b23b40-0cf4-4b1f-afaf-7bba7505c10c service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:28:19.859699 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5b23b40-0cf4-4b1f-afaf-7bba7505c10c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:28:19.861371 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-a5b23b40-0cf4-4b1f-afaf-7bba7505c10c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:28:19.906153 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:28:19.974324 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5b23b40-0cf4-4b1f-afaf-7bba7505c10c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:28:19.975756 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5b23b40-0cf4-4b1f-afaf-7bba7505c10c service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:28:19.994776 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 669/1346] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:28:19 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 317 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:28:24.168232 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae98c1c-c854-4394-b207-853015f8c705 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:28:24.168798 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae98c1c-c854-4394-b207-853015f8c705 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:28:24.168798 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae98c1c-c854-4394-b207-853015f8c705 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:28:24.170153 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 678/1347] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:28:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:28:24.175626 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4538989-f54f-420f-abaa-38f10e28c827 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:28:24.176054 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4538989-f54f-420f-abaa-38f10e28c827 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:28:24.176476 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4538989-f54f-420f-abaa-38f10e28c827 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:28:24.243322 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-e4538989-f54f-420f-abaa-38f10e28c827 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:28:24.244384 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-e4538989-f54f-420f-abaa-38f10e28c827 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:28:24.338097 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:28:24.340680 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 670/1348] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:28:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 04:28:24.627671 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ae98c1c-c854-4394-b207-853015f8c705 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:28:24.629322 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-2ae98c1c-c854-4394-b207-853015f8c705 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:28:24.735097 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ae98c1c-c854-4394-b207-853015f8c705 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:28:24.783086 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:28:24.808556 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a98db88-33bc-4589-8108-a8cc5357eecc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:28:24.809180 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a98db88-33bc-4589-8108-a8cc5357eecc service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:28:24.809457 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a98db88-33bc-4589-8108-a8cc5357eecc service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:28:24.810001 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a98db88-33bc-4589-8108-a8cc5357eecc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:28:24.811582 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-2a98db88-33bc-4589-8108-a8cc5357eecc service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:28:24.919889 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a98db88-33bc-4589-8108-a8cc5357eecc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:28:24.921350 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a98db88-33bc-4589-8108-a8cc5357eecc service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:28:24.939920 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 679/1349] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:28:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 313 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:28:29.897626 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d4c8c2c-e24f-4fda-aa5d-9db7d6a9ac69 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:28:29.898058 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d4c8c2c-e24f-4fda-aa5d-9db7d6a9ac69 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:28:29.898142 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d4c8c2c-e24f-4fda-aa5d-9db7d6a9ac69 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:28:29.900356 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 671/1350] 10.222.0.25 () {54 vars in 921 bytes} [Sun Sep 25 04:28:29 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:28:29.905625 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a3acb9-9854-44e8-9c9c-0a3897ae6d55 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:28:29.906063 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a3acb9-9854-44e8-9c9c-0a3897ae6d55 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:28:29.906489 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a3acb9-9854-44e8-9c9c-0a3897ae6d55 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:28:30.004186 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-31a3acb9-9854-44e8-9c9c-0a3897ae6d55 None None] MFA Rules not processed for user `676d04ef5a984edcaad63d9156568660`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:28:30.010808 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-31a3acb9-9854-44e8-9c9c-0a3897ae6d55 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:28:30.036133 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:28:30.156105 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 680/1351] 10.222.0.25 () {58 vars in 1018 bytes} [Sun Sep 25 04:28:29 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 251 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:28:34.510255 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3270dc9e-37b5-4dbf-ba6b-67f28e6cabbc None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:28:34.510800 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3270dc9e-37b5-4dbf-ba6b-67f28e6cabbc None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:28:34.510800 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3270dc9e-37b5-4dbf-ba6b-67f28e6cabbc None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:28:34.512320 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 672/1352] 10.222.0.25 () {54 vars in 921 bytes} [Sun Sep 25 04:28:34 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:28:34.517410 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73bc12ce-db7f-49c2-b435-525d5e55cf5c None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:28:34.518243 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73bc12ce-db7f-49c2-b435-525d5e55cf5c None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:28:34.518668 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73bc12ce-db7f-49c2-b435-525d5e55cf5c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:28:34.602814 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-73bc12ce-db7f-49c2-b435-525d5e55cf5c None None] MFA Rules not processed for user `676d04ef5a984edcaad63d9156568660`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:28:34.604440 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-73bc12ce-db7f-49c2-b435-525d5e55cf5c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:28:34.685543 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:28:34.728977 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 681/1353] 10.222.0.25 () {58 vars in 1018 bytes} [Sun Sep 25 04:28:34 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 213 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:28:36.075859 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae35ed56-e70c-4134-a698-4298fb047685 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:28:36.076420 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae35ed56-e70c-4134-a698-4298fb047685 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:28:36.076532 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae35ed56-e70c-4134-a698-4298fb047685 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:28:36.077991 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 673/1354] 10.222.0.25 () {54 vars in 921 bytes} [Sun Sep 25 04:28:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:28:36.083213 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94c4bf74-4647-4b53-bebd-41a89829e417 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:28:36.083608 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94c4bf74-4647-4b53-bebd-41a89829e417 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:28:36.083759 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94c4bf74-4647-4b53-bebd-41a89829e417 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:28:36.162667 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-94c4bf74-4647-4b53-bebd-41a89829e417 None None] MFA Rules not processed for user `676d04ef5a984edcaad63d9156568660`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:28:36.163865 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-94c4bf74-4647-4b53-bebd-41a89829e417 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:28:36.273937 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 682/1355] 10.222.0.25 () {58 vars in 1018 bytes} [Sun Sep 25 04:28:36 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 191 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:28:39.767024 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef990aad-c5f3-4db6-b93b-3df160a6c424 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:28:39.767565 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef990aad-c5f3-4db6-b93b-3df160a6c424 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:28:39.767565 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef990aad-c5f3-4db6-b93b-3df160a6c424 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:28:39.768861 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 674/1356] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:28:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:28:39.774668 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ccbe5c2-5825-475c-864d-1f7da79a3e1a None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:28:39.775121 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ccbe5c2-5825-475c-864d-1f7da79a3e1a None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:28:39.775721 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ccbe5c2-5825-475c-864d-1f7da79a3e1a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:28:39.817700 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:28:39.855241 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-3ccbe5c2-5825-475c-864d-1f7da79a3e1a None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:28:39.856493 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-3ccbe5c2-5825-475c-864d-1f7da79a3e1a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:28:39.967489 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 683/1357] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:28:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:28:40.283609 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0cefce2-60a6-4ec7-ad09-2c0fdb7fa18f None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:28:40.284801 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0cefce2-60a6-4ec7-ad09-2c0fdb7fa18f None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:28:40.285599 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0cefce2-60a6-4ec7-ad09-2c0fdb7fa18f None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:28:40.287593 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 675/1358] 10.222.0.25 () {54 vars in 961 bytes} [Sun Sep 25 04:28:40 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:28:40.298113 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cabe79b7-c463-4595-939a-b1497db84870 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:28:40.299079 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cabe79b7-c463-4595-939a-b1497db84870 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:28:40.300064 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cabe79b7-c463-4595-939a-b1497db84870 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:28:40.390144 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-cabe79b7-c463-4595-939a-b1497db84870 None None] MFA Rules not processed for user `e703b42c53974daaa608b0ccc3f018b1`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:28:40.391526 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-cabe79b7-c463-4595-939a-b1497db84870 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:28:40.447655 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:28:40.531961 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 684/1359] 10.222.0.25 () {58 vars in 1058 bytes} [Sun Sep 25 04:28:40 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 236 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:28:40.541032 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0cefce2-60a6-4ec7-ad09-2c0fdb7fa18f None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:28:40.543534 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-a0cefce2-60a6-4ec7-ad09-2c0fdb7fa18f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:28:40.705373 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0cefce2-60a6-4ec7-ad09-2c0fdb7fa18f None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:28:40.780705 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:28:40.823400 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efe12204-89d8-461b-a261-a2a9b44b8dce service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e703b42c53974daaa608b0ccc3f018b1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'ResellerAdmin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:28:40.824007 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efe12204-89d8-461b-a261-a2a9b44b8dce service nova] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:28:40.824285 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efe12204-89d8-461b-a261-a2a9b44b8dce service nova] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:28:40.824556 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efe12204-89d8-461b-a261-a2a9b44b8dce service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:28:40.826187 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-efe12204-89d8-461b-a261-a2a9b44b8dce service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:28:40.930872 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efe12204-89d8-461b-a261-a2a9b44b8dce service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:28:40.932480 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efe12204-89d8-461b-a261-a2a9b44b8dce service nova] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:28:40.951667 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 676/1360] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:28:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 411 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:28:43.366014 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc9fc1c0-e599-432f-bd59-78aea9a0cb18 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:28:43.366457 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc9fc1c0-e599-432f-bd59-78aea9a0cb18 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:28:43.367252 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc9fc1c0-e599-432f-bd59-78aea9a0cb18 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:28:43.370255 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 685/1361] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:28:43 2022] GET /identity => generated 270 bytes in 10 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:28:43.377222 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b7e96f2-3fd7-4632-a849-da0c1a0a2018 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:28:43.377611 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b7e96f2-3fd7-4632-a849-da0c1a0a2018 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:28:43.378754 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b7e96f2-3fd7-4632-a849-da0c1a0a2018 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:28:43.453116 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-4b7e96f2-3fd7-4632-a849-da0c1a0a2018 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:28:43.454750 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-4b7e96f2-3fd7-4632-a849-da0c1a0a2018 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:28:43.497787 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:28:43.560979 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 677/1362] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:28:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:28:43.856576 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc9fc1c0-e599-432f-bd59-78aea9a0cb18 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:28:43.858294 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-cc9fc1c0-e599-432f-bd59-78aea9a0cb18 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:28:44.053684 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc9fc1c0-e599-432f-bd59-78aea9a0cb18 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:28:44.066489 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:28:44.189224 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fa0967b-50f9-47ec-a544-c17391586f41 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e703b42c53974daaa608b0ccc3f018b1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'ResellerAdmin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:28:44.189895 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fa0967b-50f9-47ec-a544-c17391586f41 service nova] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:28:44.190178 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fa0967b-50f9-47ec-a544-c17391586f41 service nova] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:28:44.190450 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fa0967b-50f9-47ec-a544-c17391586f41 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:28:44.192057 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-7fa0967b-50f9-47ec-a544-c17391586f41 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:28:44.286144 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fa0967b-50f9-47ec-a544-c17391586f41 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:28:44.287758 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fa0967b-50f9-47ec-a544-c17391586f41 service nova] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:28:44.307382 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 686/1363] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:28:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 451 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:28:46.849252 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eca2379b-64c9-4746-bfe4-fc1f5e2152a4 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:28:46.849869 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eca2379b-64c9-4746-bfe4-fc1f5e2152a4 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:28:46.849869 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eca2379b-64c9-4746-bfe4-fc1f5e2152a4 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:28:46.851290 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 678/1364] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:28:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:28:46.857291 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e8d7296-f7ef-4713-9eeb-1b22448758a2 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:28:46.857742 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e8d7296-f7ef-4713-9eeb-1b22448758a2 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:28:46.857833 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e8d7296-f7ef-4713-9eeb-1b22448758a2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:28:46.873451 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:28:46.926958 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-8e8d7296-f7ef-4713-9eeb-1b22448758a2 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:28:46.928178 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-8e8d7296-f7ef-4713-9eeb-1b22448758a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:28:47.040991 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 687/1365] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:28:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:28:47.324875 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aca4607-0505-4875-a656-1d2ab57a3e3b None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:28:47.325275 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aca4607-0505-4875-a656-1d2ab57a3e3b None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:28:47.325678 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aca4607-0505-4875-a656-1d2ab57a3e3b None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:28:47.327122 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 679/1366] 10.222.0.25 () {54 vars in 961 bytes} [Sun Sep 25 04:28:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:28:47.332590 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d31217d4-0a96-4eca-bebe-577e4750116c None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:28:47.333132 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d31217d4-0a96-4eca-bebe-577e4750116c None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:28:47.333508 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d31217d4-0a96-4eca-bebe-577e4750116c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:28:47.413322 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-d31217d4-0a96-4eca-bebe-577e4750116c None None] MFA Rules not processed for user `e703b42c53974daaa608b0ccc3f018b1`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:28:47.414716 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-d31217d4-0a96-4eca-bebe-577e4750116c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:28:47.452152 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:28:47.548314 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 688/1367] 10.222.0.25 () {58 vars in 1058 bytes} [Sun Sep 25 04:28:47 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 216 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:28:47.563765 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aca4607-0505-4875-a656-1d2ab57a3e3b None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:28:47.565284 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-6aca4607-0505-4875-a656-1d2ab57a3e3b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:28:47.725624 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aca4607-0505-4875-a656-1d2ab57a3e3b None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:28:47.750847 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:28:47.853295 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-982f2d4f-c793-472e-883c-b45b4bcdaa3a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e703b42c53974daaa608b0ccc3f018b1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'ResellerAdmin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:28:47.853979 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-982f2d4f-c793-472e-883c-b45b4bcdaa3a service nova] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:28:47.854076 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-982f2d4f-c793-472e-883c-b45b4bcdaa3a service nova] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:28:47.854349 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-982f2d4f-c793-472e-883c-b45b4bcdaa3a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:28:47.855979 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-982f2d4f-c793-472e-883c-b45b4bcdaa3a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:28:47.971302 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-982f2d4f-c793-472e-883c-b45b4bcdaa3a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:28:47.973492 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-982f2d4f-c793-472e-883c-b45b4bcdaa3a service nova] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:28:48.000389 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 680/1368] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:28:47 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 437 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:28:48.027870 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2109f2b-9ac3-48ad-add0-dcc8246d4af0 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:28:48.028434 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2109f2b-9ac3-48ad-add0-dcc8246d4af0 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:28:48.028434 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2109f2b-9ac3-48ad-add0-dcc8246d4af0 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:28:48.034026 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 689/1369] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:28:48 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:28:48.035463 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cc41fba-8d0d-4887-8299-e69f5336ecdf None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:28:48.035753 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cc41fba-8d0d-4887-8299-e69f5336ecdf None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:28:48.036103 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cc41fba-8d0d-4887-8299-e69f5336ecdf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:28:48.076296 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:28:48.116212 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-1cc41fba-8d0d-4887-8299-e69f5336ecdf None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:28:48.117114 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-1cc41fba-8d0d-4887-8299-e69f5336ecdf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:28:48.248731 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 681/1370] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:28:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 211 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:28:48.582340 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbf67cb3-53ef-414f-84b2-ee033b2cd4d4 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:28:48.582746 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbf67cb3-53ef-414f-84b2-ee033b2cd4d4 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:28:48.583124 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbf67cb3-53ef-414f-84b2-ee033b2cd4d4 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:28:48.584719 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 690/1371] 10.222.0.25 () {54 vars in 963 bytes} [Sun Sep 25 04:28:48 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:28:48.592182 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bee1e788-80a6-4530-92b6-d0b073bba676 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:28:48.592580 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bee1e788-80a6-4530-92b6-d0b073bba676 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:28:48.593014 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bee1e788-80a6-4530-92b6-d0b073bba676 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:28:48.680854 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-bee1e788-80a6-4530-92b6-d0b073bba676 None None] MFA Rules not processed for user `ffbc136c77b84e8c87128f7247f4ec03`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:28:48.682191 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-bee1e788-80a6-4530-92b6-d0b073bba676 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:28:48.735464 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:28:48.776903 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 682/1372] 10.222.0.25 () {58 vars in 1060 bytes} [Sun Sep 25 04:28:48 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:28:48.778852 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbf67cb3-53ef-414f-84b2-ee033b2cd4d4 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:28:48.780376 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-fbf67cb3-53ef-414f-84b2-ee033b2cd4d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:28:48.897458 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbf67cb3-53ef-414f-84b2-ee033b2cd4d4 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:28:48.949671 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:28:48.989671 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6f38354-0163-46a1-a3ce-748ee511ae02 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ffbc136c77b84e8c87128f7247f4ec03', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:28:48.990601 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6f38354-0163-46a1-a3ce-748ee511ae02 service cinder] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:28:48.991102 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6f38354-0163-46a1-a3ce-748ee511ae02 service cinder] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:28:48.991485 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6f38354-0163-46a1-a3ce-748ee511ae02 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:28:48.994307 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-e6f38354-0163-46a1-a3ce-748ee511ae02 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:28:49.136209 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6f38354-0163-46a1-a3ce-748ee511ae02 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:28:49.137796 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6f38354-0163-46a1-a3ce-748ee511ae02 service cinder] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:28:49.157300 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 691/1373] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:28:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 379 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:28:50.310525 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a2c700f-bbfa-44e6-b3f0-6c4bdefedd34 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:28:50.311124 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a2c700f-bbfa-44e6-b3f0-6c4bdefedd34 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:28:50.311124 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a2c700f-bbfa-44e6-b3f0-6c4bdefedd34 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:28:50.312462 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 683/1374] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:28:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:28:50.318057 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe9a77e2-4f64-4613-a381-739e81ba88cc None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:28:50.318489 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe9a77e2-4f64-4613-a381-739e81ba88cc None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:28:50.318638 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe9a77e2-4f64-4613-a381-739e81ba88cc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:28:50.385440 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-fe9a77e2-4f64-4613-a381-739e81ba88cc None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:28:50.386678 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-fe9a77e2-4f64-4613-a381-739e81ba88cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:28:50.407806 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:28:50.488526 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 692/1375] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:28:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:28:50.780902 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a2c700f-bbfa-44e6-b3f0-6c4bdefedd34 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:28:50.782428 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-6a2c700f-bbfa-44e6-b3f0-6c4bdefedd34 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:28:50.934992 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a2c700f-bbfa-44e6-b3f0-6c4bdefedd34 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:28:50.988612 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:28:51.057625 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-869bbc71-7d29-46f7-85a5-69be1cdd61c0 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e703b42c53974daaa608b0ccc3f018b1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'ResellerAdmin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:28:51.058242 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-869bbc71-7d29-46f7-85a5-69be1cdd61c0 service nova] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:28:51.058524 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-869bbc71-7d29-46f7-85a5-69be1cdd61c0 service nova] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:28:51.058797 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-869bbc71-7d29-46f7-85a5-69be1cdd61c0 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:28:51.060434 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-869bbc71-7d29-46f7-85a5-69be1cdd61c0 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:28:51.159892 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-869bbc71-7d29-46f7-85a5-69be1cdd61c0 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:28:51.161541 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-869bbc71-7d29-46f7-85a5-69be1cdd61c0 service nova] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:28:51.181786 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 684/1376] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:28:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 401 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:28:53.445647 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b5c557f-d522-4296-8409-1e6da630c6fa None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:28:53.446438 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b5c557f-d522-4296-8409-1e6da630c6fa None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:28:53.446438 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b5c557f-d522-4296-8409-1e6da630c6fa None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:28:53.447710 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 693/1377] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:28:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:28:53.452914 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6baac07-9b99-4409-90a0-a46a56657847 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:28:53.453360 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6baac07-9b99-4409-90a0-a46a56657847 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:28:53.453466 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6baac07-9b99-4409-90a0-a46a56657847 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:28:53.533628 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-a6baac07-9b99-4409-90a0-a46a56657847 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:28:53.534873 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-a6baac07-9b99-4409-90a0-a46a56657847 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:28:53.543450 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:28:53.640433 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 685/1378] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:28:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:28:53.932996 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b5c557f-d522-4296-8409-1e6da630c6fa None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:28:53.935330 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-4b5c557f-d522-4296-8409-1e6da630c6fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:28:54.130056 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b5c557f-d522-4296-8409-1e6da630c6fa None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:28:54.134992 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:28:54.252687 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2780adbb-0f2c-40dc-b3f0-25013ce0a9c3 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e703b42c53974daaa608b0ccc3f018b1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'ResellerAdmin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:28:54.253337 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2780adbb-0f2c-40dc-b3f0-25013ce0a9c3 service nova] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:28:54.253622 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2780adbb-0f2c-40dc-b3f0-25013ce0a9c3 service nova] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:28:54.253896 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2780adbb-0f2c-40dc-b3f0-25013ce0a9c3 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:28:54.255483 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-2780adbb-0f2c-40dc-b3f0-25013ce0a9c3 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:28:54.353593 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2780adbb-0f2c-40dc-b3f0-25013ce0a9c3 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:28:54.355245 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2780adbb-0f2c-40dc-b3f0-25013ce0a9c3 service nova] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:28:54.375765 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 694/1379] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:28:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 444 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:28:56.619415 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2fbcead-d90e-4d99-ba8f-aebe3d69e0c4 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:28:56.619876 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2fbcead-d90e-4d99-ba8f-aebe3d69e0c4 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:28:56.620254 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2fbcead-d90e-4d99-ba8f-aebe3d69e0c4 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:28:56.622101 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 686/1380] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:28:56 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:28:56.627124 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6df8a7b9-a6ab-4498-a54c-6a11d1978cdb None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:28:56.627518 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6df8a7b9-a6ab-4498-a54c-6a11d1978cdb None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:28:56.627832 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6df8a7b9-a6ab-4498-a54c-6a11d1978cdb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:28:56.647325 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:28:56.755358 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-6df8a7b9-a6ab-4498-a54c-6a11d1978cdb None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:28:56.756591 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-6df8a7b9-a6ab-4498-a54c-6a11d1978cdb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:28:56.860826 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 695/1381] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:28:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 235 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:28:57.153214 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2fbcead-d90e-4d99-ba8f-aebe3d69e0c4 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:28:57.155043 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-a2fbcead-d90e-4d99-ba8f-aebe3d69e0c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:28:57.313488 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:28:57.325542 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2fbcead-d90e-4d99-ba8f-aebe3d69e0c4 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:28:57.447700 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2318948e-3f08-4509-b22c-51ff7f063a67 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e703b42c53974daaa608b0ccc3f018b1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'ResellerAdmin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:28:57.448242 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2318948e-3f08-4509-b22c-51ff7f063a67 service nova] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:28:57.448609 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2318948e-3f08-4509-b22c-51ff7f063a67 service nova] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:28:57.448824 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2318948e-3f08-4509-b22c-51ff7f063a67 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:28:57.450519 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-2318948e-3f08-4509-b22c-51ff7f063a67 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:28:57.547777 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2318948e-3f08-4509-b22c-51ff7f063a67 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:28:57.549608 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2318948e-3f08-4509-b22c-51ff7f063a67 service nova] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:28:57.560269 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:28:57.569183 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 687/1382] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:28:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 417 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:28:59.831036 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76b40499-f56b-46bb-9d5e-4a6f537061d2 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:28:59.831717 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76b40499-f56b-46bb-9d5e-4a6f537061d2 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:28:59.831717 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76b40499-f56b-46bb-9d5e-4a6f537061d2 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:28:59.832959 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 696/1383] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:28:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:28:59.839400 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15ad026f-b032-469a-8a6d-86d84aa82635 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:28:59.839860 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15ad026f-b032-469a-8a6d-86d84aa82635 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:28:59.839981 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15ad026f-b032-469a-8a6d-86d84aa82635 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:28:59.916325 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-15ad026f-b032-469a-8a6d-86d84aa82635 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:28:59.917614 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-15ad026f-b032-469a-8a6d-86d84aa82635 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:29:00.018441 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 688/1384] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:28:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:29:00.308107 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76b40499-f56b-46bb-9d5e-4a6f537061d2 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:29:00.310221 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-76b40499-f56b-46bb-9d5e-4a6f537061d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:29:00.404615 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:29:00.479341 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76b40499-f56b-46bb-9d5e-4a6f537061d2 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:29:00.598925 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02826a07-70ee-4cfd-9929-847b297f66b1 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e703b42c53974daaa608b0ccc3f018b1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'ResellerAdmin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:29:00.599550 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02826a07-70ee-4cfd-9929-847b297f66b1 service nova] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:29:00.599836 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02826a07-70ee-4cfd-9929-847b297f66b1 service nova] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:29:00.600117 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02826a07-70ee-4cfd-9929-847b297f66b1 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:29:00.602378 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-02826a07-70ee-4cfd-9929-847b297f66b1 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:29:00.654629 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:29:00.698629 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02826a07-70ee-4cfd-9929-847b297f66b1 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:29:00.700233 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02826a07-70ee-4cfd-9929-847b297f66b1 service nova] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:29:00.719986 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 697/1385] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:29:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 413 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:29:02.917552 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa669d4-6ab9-4afd-b713-ce9f44c32b4a None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:29:02.918129 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa669d4-6ab9-4afd-b713-ce9f44c32b4a None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:29:02.918129 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa669d4-6ab9-4afd-b713-ce9f44c32b4a None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:29:02.919599 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 689/1386] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:29:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:29:02.924813 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e67e6a2-a387-41f4-a257-50dd77e85a56 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:29:02.925245 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e67e6a2-a387-41f4-a257-50dd77e85a56 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:29:02.925354 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e67e6a2-a387-41f4-a257-50dd77e85a56 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:29:02.993008 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-1e67e6a2-a387-41f4-a257-50dd77e85a56 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:29:02.994252 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-1e67e6a2-a387-41f4-a257-50dd77e85a56 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:29:03.095133 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 698/1387] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:29:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:29:03.387155 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baa669d4-6ab9-4afd-b713-ce9f44c32b4a None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:29:03.389446 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-baa669d4-6ab9-4afd-b713-ce9f44c32b4a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:29:03.472079 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:29:03.547614 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baa669d4-6ab9-4afd-b713-ce9f44c32b4a None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:29:03.662702 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-899a1345-a001-4def-9dad-a928beb74beb service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e703b42c53974daaa608b0ccc3f018b1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'ResellerAdmin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:29:03.663322 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-899a1345-a001-4def-9dad-a928beb74beb service nova] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:29:03.663651 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-899a1345-a001-4def-9dad-a928beb74beb service nova] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:29:03.664006 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-899a1345-a001-4def-9dad-a928beb74beb service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:29:03.665652 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-899a1345-a001-4def-9dad-a928beb74beb service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:29:03.706097 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:29:03.759493 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-899a1345-a001-4def-9dad-a928beb74beb service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:29:03.761142 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-899a1345-a001-4def-9dad-a928beb74beb service nova] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:29:03.779932 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 690/1388] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:29:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 393 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:29:04.522333 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b98cddb-7aa5-46e2-8b29-1d814b870dcc None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:29:04.522972 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b98cddb-7aa5-46e2-8b29-1d814b870dcc None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:29:04.523137 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b98cddb-7aa5-46e2-8b29-1d814b870dcc None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:29:04.524572 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 699/1389] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:29:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:29:04.530116 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f518378-6900-475a-81d3-c3fe06a5cb63 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:29:04.530549 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f518378-6900-475a-81d3-c3fe06a5cb63 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:29:04.530680 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f518378-6900-475a-81d3-c3fe06a5cb63 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:29:04.598740 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-5f518378-6900-475a-81d3-c3fe06a5cb63 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:29:04.599970 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-5f518378-6900-475a-81d3-c3fe06a5cb63 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:29:04.699579 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 691/1390] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:29:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:29:05.000923 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da49b698-9c5c-4fee-ac4b-7d6239b24882 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:29:05.001211 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da49b698-9c5c-4fee-ac4b-7d6239b24882 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:29:05.001487 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da49b698-9c5c-4fee-ac4b-7d6239b24882 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:29:05.002882 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 700/1391] 10.222.0.25 () {54 vars in 962 bytes} [Sun Sep 25 04:29:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:29:05.009029 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c69f95f2-1e64-4d90-a23e-f3cff5c66e4a None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:29:05.009318 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c69f95f2-1e64-4d90-a23e-f3cff5c66e4a None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:29:05.009594 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c69f95f2-1e64-4d90-a23e-f3cff5c66e4a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:29:05.025859 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:29:05.089132 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-c69f95f2-1e64-4d90-a23e-f3cff5c66e4a None None] MFA Rules not processed for user `316eaa67d5464091af47efd489b96b87`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:29:05.090348 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-c69f95f2-1e64-4d90-a23e-f3cff5c66e4a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:29:05.168303 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 692/1392] 10.222.0.25 () {58 vars in 1059 bytes} [Sun Sep 25 04:29:05 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:29:05.177427 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da49b698-9c5c-4fee-ac4b-7d6239b24882 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:29:05.179167 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-da49b698-9c5c-4fee-ac4b-7d6239b24882 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:29:05.211823 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:29:05.284311 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da49b698-9c5c-4fee-ac4b-7d6239b24882 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:29:05.357782 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30c7c827-0ea9-4ede-a29f-af5ce4e90650 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '316eaa67d5464091af47efd489b96b87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:29:05.358395 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30c7c827-0ea9-4ede-a29f-af5ce4e90650 service glance] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:29:05.358676 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30c7c827-0ea9-4ede-a29f-af5ce4e90650 service glance] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:29:05.358967 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30c7c827-0ea9-4ede-a29f-af5ce4e90650 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:29:05.360545 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-30c7c827-0ea9-4ede-a29f-af5ce4e90650 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:29:05.439649 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:29:05.473578 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30c7c827-0ea9-4ede-a29f-af5ce4e90650 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:29:05.475048 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30c7c827-0ea9-4ede-a29f-af5ce4e90650 service glance] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:29:05.494531 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 701/1393] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:29:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 318 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:29:05.993594 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca84c4eb-cb62-40da-a1f5-594d450d5b91 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:29:05.994116 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca84c4eb-cb62-40da-a1f5-594d450d5b91 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:29:05.994116 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca84c4eb-cb62-40da-a1f5-594d450d5b91 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:29:05.995635 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 693/1394] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:29:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:29:06.000936 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a795c8-353c-46e1-83e1-472a36c9177e None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:29:06.001402 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a795c8-353c-46e1-83e1-472a36c9177e None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:29:06.001503 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a795c8-353c-46e1-83e1-472a36c9177e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:29:06.072727 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-55a795c8-353c-46e1-83e1-472a36c9177e None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:29:06.074008 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-55a795c8-353c-46e1-83e1-472a36c9177e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:29:06.180939 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 702/1395] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:29:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:29:06.249781 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f064f43a-bacd-4fc5-8a4e-7291826d3566 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:29:06.250077 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f064f43a-bacd-4fc5-8a4e-7291826d3566 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:29:06.250418 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f064f43a-bacd-4fc5-8a4e-7291826d3566 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:29:06.318535 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-f064f43a-bacd-4fc5-8a4e-7291826d3566 None None] MFA Rules not processed for user `050b3aab44bb4b17aa4a1ba7cccdb231`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:29:06.320505 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-f064f43a-bacd-4fc5-8a4e-7291826d3566 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:29:06.374570 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:29:06.406926 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 694/1396] 10.222.0.25 () {58 vars in 1009 bytes} [Sun Sep 25 04:29:06 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:29:06.452908 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-635a9b89-e8fe-47ce-985b-4dd84b0610d9 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:29:06.453199 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-635a9b89-e8fe-47ce-985b-4dd84b0610d9 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:29:06.453476 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-635a9b89-e8fe-47ce-985b-4dd84b0610d9 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:29:06.454817 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 703/1397] 10.222.0.25 () {54 vars in 943 bytes} [Sun Sep 25 04:29:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:29:06.462182 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-795db70f-3d1e-4a91-9b59-0b5f03001df6 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:29:06.462476 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-795db70f-3d1e-4a91-9b59-0b5f03001df6 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:29:06.462754 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-795db70f-3d1e-4a91-9b59-0b5f03001df6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:29:06.476925 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-635a9b89-e8fe-47ce-985b-4dd84b0610d9 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:29:06.478447 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-635a9b89-e8fe-47ce-985b-4dd84b0610d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:29:06.520313 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:29:06.545085 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-795db70f-3d1e-4a91-9b59-0b5f03001df6 None None] MFA Rules not processed for user `23ea88f46e7740d295148a93ade29dd9`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:29:06.546320 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-795db70f-3d1e-4a91-9b59-0b5f03001df6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:29:06.630058 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-635a9b89-e8fe-47ce-985b-4dd84b0610d9 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:29:06.655398 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 695/1398] 10.222.0.25 () {58 vars in 1040 bytes} [Sun Sep 25 04:29:06 2022] POST /identity/v3/auth/tokens => generated 3304 bytes in 195 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:29:06.667936 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-795db70f-3d1e-4a91-9b59-0b5f03001df6 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:29:06.677634 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-795db70f-3d1e-4a91-9b59-0b5f03001df6 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:29:06.707115 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:29:06.760089 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:29:06.762234 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-820c6d43-50d1-4db6-9fe4-fb0aeed9eba7 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e703b42c53974daaa608b0ccc3f018b1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'ResellerAdmin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:29:06.763126 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-820c6d43-50d1-4db6-9fe4-fb0aeed9eba7 service nova] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:29:06.763739 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-820c6d43-50d1-4db6-9fe4-fb0aeed9eba7 service nova] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:29:06.764234 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-820c6d43-50d1-4db6-9fe4-fb0aeed9eba7 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:29:06.766311 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-820c6d43-50d1-4db6-9fe4-fb0aeed9eba7 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:29:06.804733 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7977457e-f466-4116-a909-c45d8284571b service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '23ea88f46e7740d295148a93ade29dd9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:29:06.805466 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7977457e-f466-4116-a909-c45d8284571b service swift] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:29:06.805806 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7977457e-f466-4116-a909-c45d8284571b service swift] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:29:06.806145 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7977457e-f466-4116-a909-c45d8284571b service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:29:06.807885 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-7977457e-f466-4116-a909-c45d8284571b service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:29:06.875849 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-820c6d43-50d1-4db6-9fe4-fb0aeed9eba7 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:29:06.877497 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-820c6d43-50d1-4db6-9fe4-fb0aeed9eba7 service nova] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:29:06.896515 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 704/1399] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:29:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 420 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:29:06.906925 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7977457e-f466-4116-a909-c45d8284571b service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:29:06.908562 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7977457e-f466-4116-a909-c45d8284571b service swift] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:29:06.917597 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 696/1400] 10.222.0.25 () {60 vars in 1384 bytes} [Sun Sep 25 04:29:06 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 250 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 25 04:29:09.135353 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1ca55e2-5581-424d-a56d-d1c5b7521841 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:29:09.135884 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1ca55e2-5581-424d-a56d-d1c5b7521841 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:29:09.135884 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1ca55e2-5581-424d-a56d-d1c5b7521841 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:29:09.137479 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 705/1401] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:29:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:29:09.142610 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-141a1fdb-1864-4cd1-94e9-d16a516522fe None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:29:09.143061 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-141a1fdb-1864-4cd1-94e9-d16a516522fe None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:29:09.143192 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-141a1fdb-1864-4cd1-94e9-d16a516522fe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:29:09.221082 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-141a1fdb-1864-4cd1-94e9-d16a516522fe None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:29:09.222547 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-141a1fdb-1864-4cd1-94e9-d16a516522fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:29:09.229950 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:29:09.336542 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 697/1402] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:29:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 195 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:29:09.628119 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1ca55e2-5581-424d-a56d-d1c5b7521841 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:29:09.629694 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-b1ca55e2-5581-424d-a56d-d1c5b7521841 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:29:09.764140 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:29:09.797504 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1ca55e2-5581-424d-a56d-d1c5b7521841 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:29:09.925828 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca44c59e-9bde-4e2b-9eee-966a4d03bce3 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e703b42c53974daaa608b0ccc3f018b1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'ResellerAdmin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:29:09.926448 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca44c59e-9bde-4e2b-9eee-966a4d03bce3 service nova] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:29:09.926725 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca44c59e-9bde-4e2b-9eee-966a4d03bce3 service nova] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:29:09.926983 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca44c59e-9bde-4e2b-9eee-966a4d03bce3 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:29:09.928599 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-ca44c59e-9bde-4e2b-9eee-966a4d03bce3 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:29:10.017117 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:29:10.029559 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca44c59e-9bde-4e2b-9eee-966a4d03bce3 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:29:10.029945 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca44c59e-9bde-4e2b-9eee-966a4d03bce3 service nova] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:29:10.050369 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 706/1403] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:29:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 422 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:29:12.307959 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b0348c2-52bd-43a3-96f7-29df1c383be8 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:29:12.308571 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b0348c2-52bd-43a3-96f7-29df1c383be8 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:29:12.308571 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b0348c2-52bd-43a3-96f7-29df1c383be8 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:29:12.310041 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 698/1404] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:29:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:29:12.315248 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc2ccef2-1a4f-40c2-a51a-f92d918429a0 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:29:12.315689 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc2ccef2-1a4f-40c2-a51a-f92d918429a0 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:29:12.315782 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc2ccef2-1a4f-40c2-a51a-f92d918429a0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:29:12.385221 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-cc2ccef2-1a4f-40c2-a51a-f92d918429a0 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:29:12.386548 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-cc2ccef2-1a4f-40c2-a51a-f92d918429a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:29:12.516815 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 707/1405] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:29:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 199 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:29:12.815196 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b0348c2-52bd-43a3-96f7-29df1c383be8 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:29:12.817207 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-7b0348c2-52bd-43a3-96f7-29df1c383be8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:29:12.976308 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:29:12.987615 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b0348c2-52bd-43a3-96f7-29df1c383be8 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:29:13.120572 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe3bbead-c74b-430a-8dea-26563c986c81 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e703b42c53974daaa608b0ccc3f018b1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'ResellerAdmin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:29:13.121219 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe3bbead-c74b-430a-8dea-26563c986c81 service nova] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:29:13.121504 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe3bbead-c74b-430a-8dea-26563c986c81 service nova] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:29:13.121779 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe3bbead-c74b-430a-8dea-26563c986c81 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:29:13.123413 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-fe3bbead-c74b-430a-8dea-26563c986c81 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:29:13.223943 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe3bbead-c74b-430a-8dea-26563c986c81 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:29:13.225672 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe3bbead-c74b-430a-8dea-26563c986c81 service nova] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:29:13.235949 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:29:13.245211 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 699/1406] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:29:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 431 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:29:15.471186 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-031c7d5a-198b-49be-987a-f2d623bcd51f None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:29:15.471781 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-031c7d5a-198b-49be-987a-f2d623bcd51f None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:29:15.471781 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-031c7d5a-198b-49be-987a-f2d623bcd51f None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:29:15.473452 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 708/1407] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:29:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:29:15.481139 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2daf6837-b3c3-4e8e-a83e-9d4b7919de6e None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:29:15.481697 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2daf6837-b3c3-4e8e-a83e-9d4b7919de6e None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:29:15.481697 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2daf6837-b3c3-4e8e-a83e-9d4b7919de6e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:29:15.558447 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-2daf6837-b3c3-4e8e-a83e-9d4b7919de6e None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:29:15.559684 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-2daf6837-b3c3-4e8e-a83e-9d4b7919de6e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:29:15.667727 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 700/1408] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:29:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:29:15.962769 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-031c7d5a-198b-49be-987a-f2d623bcd51f None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:29:15.964533 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-031c7d5a-198b-49be-987a-f2d623bcd51f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:29:16.035906 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:29:16.129257 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-031c7d5a-198b-49be-987a-f2d623bcd51f None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:29:16.258658 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5823665f-41f1-4299-b70f-963b937d0a71 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e703b42c53974daaa608b0ccc3f018b1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'ResellerAdmin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:29:16.259266 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5823665f-41f1-4299-b70f-963b937d0a71 service nova] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:29:16.259547 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5823665f-41f1-4299-b70f-963b937d0a71 service nova] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:29:16.259826 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5823665f-41f1-4299-b70f-963b937d0a71 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:29:16.261470 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-5823665f-41f1-4299-b70f-963b937d0a71 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:29:16.284023 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:29:16.359385 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5823665f-41f1-4299-b70f-963b937d0a71 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:29:16.361140 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5823665f-41f1-4299-b70f-963b937d0a71 service nova] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:29:16.380521 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 709/1409] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:29:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 418 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:29:25.666326 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=67 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:29:26.378184 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=35 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:29:29.731855 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e49c9020-a44b-406c-9202-90529c069f44 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:29:29.732392 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e49c9020-a44b-406c-9202-90529c069f44 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:29:29.732392 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e49c9020-a44b-406c-9202-90529c069f44 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:29:29.733730 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 701/1410] 10.222.0.25 () {54 vars in 918 bytes} [Sun Sep 25 04:29:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:29:29.738904 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3cdb92f-18d4-4545-a928-56c89ee83528 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:29:29.739309 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3cdb92f-18d4-4545-a928-56c89ee83528 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:29:29.739457 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3cdb92f-18d4-4545-a928-56c89ee83528 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:29:29.814920 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-f3cdb92f-18d4-4545-a928-56c89ee83528 None None] MFA Rules not processed for user `676d04ef5a984edcaad63d9156568660`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:29:29.816123 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-f3cdb92f-18d4-4545-a928-56c89ee83528 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:29:29.920834 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 710/1411] 10.222.0.25 () {58 vars in 1015 bytes} [Sun Sep 25 04:29:29 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:29:29.935560 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d8ec496-52cb-479c-bb92-46a644231966 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:29:29.935847 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d8ec496-52cb-479c-bb92-46a644231966 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:29:29.936127 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d8ec496-52cb-479c-bb92-46a644231966 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:29:29.937513 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 702/1412] 10.222.0.25 () {54 vars in 918 bytes} [Sun Sep 25 04:29:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:29:29.942249 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19e7fdbb-6ace-48b7-9dd0-c3b686e4689f None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:29:29.942535 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19e7fdbb-6ace-48b7-9dd0-c3b686e4689f None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:29:29.942807 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19e7fdbb-6ace-48b7-9dd0-c3b686e4689f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:29:30.014906 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:29:30.017856 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-19e7fdbb-6ace-48b7-9dd0-c3b686e4689f None None] MFA Rules not processed for user `e703b42c53974daaa608b0ccc3f018b1`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:29:30.019658 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-19e7fdbb-6ace-48b7-9dd0-c3b686e4689f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:29:30.134423 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 711/1413] 10.222.0.25 () {58 vars in 1015 bytes} [Sun Sep 25 04:29:29 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 193 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:29:32.726938 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff8bf4c-ce43-438c-a8e9-5cd1586a2b63 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:29:32.727451 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff8bf4c-ce43-438c-a8e9-5cd1586a2b63 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:29:32.727568 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff8bf4c-ce43-438c-a8e9-5cd1586a2b63 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:29:32.728916 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 703/1414] 127.0.0.1 () {54 vars in 917 bytes} [Sun Sep 25 04:29:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 25 04:29:32.734100 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c7198fe-59a6-4803-b855-8119c2921801 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:29:32.734548 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c7198fe-59a6-4803-b855-8119c2921801 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:29:32.734644 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c7198fe-59a6-4803-b855-8119c2921801 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:29:32.800737 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-3c7198fe-59a6-4803-b855-8119c2921801 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:29:32.801950 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-3c7198fe-59a6-4803-b855-8119c2921801 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:29:32.856388 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:29:32.898357 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 712/1415] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Sep 25 04:29:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:29:33.187250 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ff8bf4c-ce43-438c-a8e9-5cd1586a2b63 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:29:33.188836 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-4ff8bf4c-ce43-438c-a8e9-5cd1586a2b63 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:29:33.333207 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ff8bf4c-ce43-438c-a8e9-5cd1586a2b63 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:29:33.406617 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:29:33.447625 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-737c4a70-9841-42fc-91d2-9f1045388222 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e703b42c53974daaa608b0ccc3f018b1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'ResellerAdmin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:29:33.448157 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-737c4a70-9841-42fc-91d2-9f1045388222 service nova] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:29:33.448436 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-737c4a70-9841-42fc-91d2-9f1045388222 service nova] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:29:33.448736 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-737c4a70-9841-42fc-91d2-9f1045388222 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:29:33.450323 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-737c4a70-9841-42fc-91d2-9f1045388222 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:29:33.538964 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-737c4a70-9841-42fc-91d2-9f1045388222 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:29:33.540862 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-737c4a70-9841-42fc-91d2-9f1045388222 service nova] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:29:33.558154 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 704/1416] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:29:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 372 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:29:34.823487 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba31aa3-40a1-4e34-9428-d4ef69224c17 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:29:34.824013 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba31aa3-40a1-4e34-9428-d4ef69224c17 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:29:34.824013 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba31aa3-40a1-4e34-9428-d4ef69224c17 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:29:34.825978 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 713/1417] 127.0.0.1 () {54 vars in 901 bytes} [Sun Sep 25 04:29:34 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:29:34.836135 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2de01e22-f922-4551-8a9e-eb7a51611c74 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:29:34.836958 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2de01e22-f922-4551-8a9e-eb7a51611c74 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:29:34.836958 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2de01e22-f922-4551-8a9e-eb7a51611c74 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:29:34.926949 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-2de01e22-f922-4551-8a9e-eb7a51611c74 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:29:34.928283 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-2de01e22-f922-4551-8a9e-eb7a51611c74 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:29:34.954427 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:29:35.031716 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 705/1418] 127.0.0.1 () {58 vars in 998 bytes} [Sun Sep 25 04:29:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 198 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:29:35.187535 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ba31aa3-40a1-4e34-9428-d4ef69224c17 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:29:35.189158 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-0ba31aa3-40a1-4e34-9428-d4ef69224c17 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:29:35.338950 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ba31aa3-40a1-4e34-9428-d4ef69224c17 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:29:35.379282 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:29:35.452923 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b4cdfd4-5989-4679-b295-1abd3cbad3a3 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e703b42c53974daaa608b0ccc3f018b1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'ResellerAdmin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:29:35.453447 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b4cdfd4-5989-4679-b295-1abd3cbad3a3 service nova] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:29:35.453721 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b4cdfd4-5989-4679-b295-1abd3cbad3a3 service nova] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:29:35.453988 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b4cdfd4-5989-4679-b295-1abd3cbad3a3 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:29:35.455811 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-7b4cdfd4-5989-4679-b295-1abd3cbad3a3 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:29:35.544575 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b4cdfd4-5989-4679-b295-1abd3cbad3a3 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:29:35.546214 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b4cdfd4-5989-4679-b295-1abd3cbad3a3 service nova] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:29:35.564493 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 714/1419] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:29:35 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 377 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:29:36.715634 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4eb407c-6931-495b-ad3c-d2e30cf185b8 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:29:36.716179 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4eb407c-6931-495b-ad3c-d2e30cf185b8 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:29:36.716179 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4eb407c-6931-495b-ad3c-d2e30cf185b8 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:29:36.717490 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 706/1420] 127.0.0.1 () {54 vars in 901 bytes} [Sun Sep 25 04:29:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:29:36.723753 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3976281b-6394-4392-9ae9-0953ac9b8666 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:29:36.724171 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3976281b-6394-4392-9ae9-0953ac9b8666 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:29:36.724304 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3976281b-6394-4392-9ae9-0953ac9b8666 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:29:36.782634 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:29:36.805250 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-3976281b-6394-4392-9ae9-0953ac9b8666 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:29:36.806463 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-3976281b-6394-4392-9ae9-0953ac9b8666 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:29:36.902865 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 715/1421] 127.0.0.1 () {58 vars in 998 bytes} [Sun Sep 25 04:29:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:29:37.064296 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4eb407c-6931-495b-ad3c-d2e30cf185b8 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:29:37.065762 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-a4eb407c-6931-495b-ad3c-d2e30cf185b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:29:37.210259 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4eb407c-6931-495b-ad3c-d2e30cf185b8 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:29:37.222017 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:29:37.321107 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67c8fc1b-9020-4b29-968f-2892bc261c21 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e703b42c53974daaa608b0ccc3f018b1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'ResellerAdmin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:29:37.321628 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c8fc1b-9020-4b29-968f-2892bc261c21 service nova] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:29:37.321906 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c8fc1b-9020-4b29-968f-2892bc261c21 service nova] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:29:37.322176 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c8fc1b-9020-4b29-968f-2892bc261c21 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:29:37.323758 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-67c8fc1b-9020-4b29-968f-2892bc261c21 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:29:37.411556 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67c8fc1b-9020-4b29-968f-2892bc261c21 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:29:37.413763 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67c8fc1b-9020-4b29-968f-2892bc261c21 service nova] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:29:37.431939 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 707/1422] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:29:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 368 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:29:38.558484 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a4e72a-d735-4611-b9ae-a02e08a60b35 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:29:38.559026 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a4e72a-d735-4611-b9ae-a02e08a60b35 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:29:38.559026 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a4e72a-d735-4611-b9ae-a02e08a60b35 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:29:38.560380 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 716/1423] 127.0.0.1 () {54 vars in 901 bytes} [Sun Sep 25 04:29:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:29:38.566577 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92266f32-cd87-472e-be04-15350c9f9d16 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:29:38.567013 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92266f32-cd87-472e-be04-15350c9f9d16 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:29:38.567126 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92266f32-cd87-472e-be04-15350c9f9d16 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:29:38.587677 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:29:38.644094 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-92266f32-cd87-472e-be04-15350c9f9d16 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:29:38.645384 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-92266f32-cd87-472e-be04-15350c9f9d16 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:29:38.741618 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 708/1424] 127.0.0.1 () {58 vars in 998 bytes} [Sun Sep 25 04:29:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:29:38.899490 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59a4e72a-d735-4611-b9ae-a02e08a60b35 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:29:38.901073 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-59a4e72a-d735-4611-b9ae-a02e08a60b35 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:29:39.015276 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:29:39.051243 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59a4e72a-d735-4611-b9ae-a02e08a60b35 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:29:39.164875 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4315d2f4-1afc-44c3-99a8-1a9686ff874f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e703b42c53974daaa608b0ccc3f018b1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'ResellerAdmin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:29:39.165485 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4315d2f4-1afc-44c3-99a8-1a9686ff874f service nova] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:29:39.165762 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4315d2f4-1afc-44c3-99a8-1a9686ff874f service nova] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:29:39.166033 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4315d2f4-1afc-44c3-99a8-1a9686ff874f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:29:39.167643 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-4315d2f4-1afc-44c3-99a8-1a9686ff874f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:29:39.245478 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:29:39.259559 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4315d2f4-1afc-44c3-99a8-1a9686ff874f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:29:39.261212 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4315d2f4-1afc-44c3-99a8-1a9686ff874f service nova] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:29:39.280317 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 717/1425] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:29:38 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 381 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:29:40.409416 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e369ff58-ef52-4ad1-9d30-b06796035faa None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:29:40.409943 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e369ff58-ef52-4ad1-9d30-b06796035faa None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:29:40.410063 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e369ff58-ef52-4ad1-9d30-b06796035faa None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:29:40.411337 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 709/1426] 127.0.0.1 () {54 vars in 901 bytes} [Sun Sep 25 04:29:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:29:40.417629 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3130d0a4-1ba3-4c98-82c1-2b10576b8854 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:29:40.418105 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3130d0a4-1ba3-4c98-82c1-2b10576b8854 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:29:40.418217 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3130d0a4-1ba3-4c98-82c1-2b10576b8854 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:29:40.495993 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-3130d0a4-1ba3-4c98-82c1-2b10576b8854 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:29:40.497251 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-3130d0a4-1ba3-4c98-82c1-2b10576b8854 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:29:40.593402 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 718/1427] 127.0.0.1 () {58 vars in 998 bytes} [Sun Sep 25 04:29:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:29:40.747845 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e369ff58-ef52-4ad1-9d30-b06796035faa None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:29:40.749313 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-e369ff58-ef52-4ad1-9d30-b06796035faa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:29:40.835163 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:29:40.898159 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e369ff58-ef52-4ad1-9d30-b06796035faa None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:29:41.014896 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fef795e-b225-493e-820b-da5abf56fc94 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e703b42c53974daaa608b0ccc3f018b1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'ResellerAdmin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:29:41.015429 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fef795e-b225-493e-820b-da5abf56fc94 service nova] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:29:41.015710 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fef795e-b225-493e-820b-da5abf56fc94 service nova] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:29:41.015980 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fef795e-b225-493e-820b-da5abf56fc94 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:29:41.017625 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-5fef795e-b225-493e-820b-da5abf56fc94 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:29:41.069862 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:29:41.112055 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fef795e-b225-493e-820b-da5abf56fc94 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:29:41.113691 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fef795e-b225-493e-820b-da5abf56fc94 service nova] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:29:41.132215 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 710/1428] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:29:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 385 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 25 04:29:42.185700 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-906b507b-d6fa-42bc-b85c-1ff6250d7748 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:29:42.186273 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-906b507b-d6fa-42bc-b85c-1ff6250d7748 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:29:42.186273 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-906b507b-d6fa-42bc-b85c-1ff6250d7748 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:29:42.187655 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 719/1429] 127.0.0.1 () {54 vars in 901 bytes} [Sun Sep 25 04:29:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:29:42.194093 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97a12734-3b58-48c7-905e-5fcfb656bc1e None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:29:42.194552 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97a12734-3b58-48c7-905e-5fcfb656bc1e None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:29:42.194664 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97a12734-3b58-48c7-905e-5fcfb656bc1e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:29:42.269536 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-97a12734-3b58-48c7-905e-5fcfb656bc1e None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:29:42.270750 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-97a12734-3b58-48c7-905e-5fcfb656bc1e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:29:42.367068 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 711/1430] 127.0.0.1 () {58 vars in 998 bytes} [Sun Sep 25 04:29:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:29:42.517706 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-906b507b-d6fa-42bc-b85c-1ff6250d7748 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:29:42.519248 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-906b507b-d6fa-42bc-b85c-1ff6250d7748 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:29:42.568924 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:29:42.666546 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-906b507b-d6fa-42bc-b85c-1ff6250d7748 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:29:42.781168 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be6b80b0-2801-4f9d-8c4a-b4b110bf2836 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e703b42c53974daaa608b0ccc3f018b1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'ResellerAdmin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:29:42.781585 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be6b80b0-2801-4f9d-8c4a-b4b110bf2836 service nova] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:29:42.781862 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be6b80b0-2801-4f9d-8c4a-b4b110bf2836 service nova] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:29:42.782135 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be6b80b0-2801-4f9d-8c4a-b4b110bf2836 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:29:42.783762 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-be6b80b0-2801-4f9d-8c4a-b4b110bf2836 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:29:42.791524 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:29:42.875155 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be6b80b0-2801-4f9d-8c4a-b4b110bf2836 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:29:42.876792 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be6b80b0-2801-4f9d-8c4a-b4b110bf2836 service nova] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:29:42.894540 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 720/1431] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:29:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 377 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:29:43.952234 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-652e6b07-6071-43fb-8579-e395a7874978 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:29:43.952836 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-652e6b07-6071-43fb-8579-e395a7874978 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:29:43.952836 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-652e6b07-6071-43fb-8579-e395a7874978 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:29:43.954180 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 712/1432] 127.0.0.1 () {54 vars in 901 bytes} [Sun Sep 25 04:29:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:29:43.961375 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5388fbf-6d92-46eb-b629-7219454837d5 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:29:43.961806 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5388fbf-6d92-46eb-b629-7219454837d5 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:29:43.961911 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5388fbf-6d92-46eb-b629-7219454837d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:29:44.038592 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-a5388fbf-6d92-46eb-b629-7219454837d5 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:29:44.039784 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-a5388fbf-6d92-46eb-b629-7219454837d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:29:44.225437 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:29:44.253959 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 721/1433] 127.0.0.1 () {58 vars in 998 bytes} [Sun Sep 25 04:29:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 294 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:29:44.404554 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-652e6b07-6071-43fb-8579-e395a7874978 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:29:44.406110 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-652e6b07-6071-43fb-8579-e395a7874978 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:29:44.423866 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:29:44.555121 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-652e6b07-6071-43fb-8579-e395a7874978 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:29:44.640490 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:29:44.667106 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f999daa-95a9-424d-a3ff-4b0702e91161 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e703b42c53974daaa608b0ccc3f018b1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'ResellerAdmin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:29:44.667728 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f999daa-95a9-424d-a3ff-4b0702e91161 service nova] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:29:44.668040 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f999daa-95a9-424d-a3ff-4b0702e91161 service nova] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:29:44.668345 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f999daa-95a9-424d-a3ff-4b0702e91161 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:29:44.670031 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-4f999daa-95a9-424d-a3ff-4b0702e91161 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:29:44.759469 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f999daa-95a9-424d-a3ff-4b0702e91161 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:29:44.761067 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f999daa-95a9-424d-a3ff-4b0702e91161 service nova] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:29:44.779888 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 713/1434] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:29:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 376 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:29:45.840861 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a92a96d-c643-4920-9b5b-dac91d2e0885 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:29:45.841404 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a92a96d-c643-4920-9b5b-dac91d2e0885 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:29:45.841404 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a92a96d-c643-4920-9b5b-dac91d2e0885 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:29:45.842630 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 722/1435] 127.0.0.1 () {54 vars in 901 bytes} [Sun Sep 25 04:29:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:29:45.849470 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d11a00bb-503e-4da1-8608-73a02624a085 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:29:45.849915 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d11a00bb-503e-4da1-8608-73a02624a085 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:29:45.850405 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d11a00bb-503e-4da1-8608-73a02624a085 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:29:45.926323 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-d11a00bb-503e-4da1-8608-73a02624a085 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:29:45.928269 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-d11a00bb-503e-4da1-8608-73a02624a085 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:29:45.969518 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:29:46.024415 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 714/1436] 127.0.0.1 () {58 vars in 998 bytes} [Sun Sep 25 04:29:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:29:46.176013 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a92a96d-c643-4920-9b5b-dac91d2e0885 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:29:46.178360 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-6a92a96d-c643-4920-9b5b-dac91d2e0885 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:29:46.320656 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a92a96d-c643-4920-9b5b-dac91d2e0885 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:29:46.375030 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:29:46.432108 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58d1dea9-7f00-4b1e-97ec-e06d43360734 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e703b42c53974daaa608b0ccc3f018b1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'ResellerAdmin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:29:46.432660 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58d1dea9-7f00-4b1e-97ec-e06d43360734 service nova] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:29:46.432942 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58d1dea9-7f00-4b1e-97ec-e06d43360734 service nova] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:29:46.433211 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58d1dea9-7f00-4b1e-97ec-e06d43360734 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:29:46.434797 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-58d1dea9-7f00-4b1e-97ec-e06d43360734 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:29:46.523804 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58d1dea9-7f00-4b1e-97ec-e06d43360734 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:29:46.525430 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58d1dea9-7f00-4b1e-97ec-e06d43360734 service nova] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:29:46.542987 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 723/1437] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:29:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 367 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:29:47.594781 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31b5c964-d25c-44ce-8b00-d6600da6e339 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:29:47.595362 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31b5c964-d25c-44ce-8b00-d6600da6e339 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:29:47.595362 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31b5c964-d25c-44ce-8b00-d6600da6e339 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:29:47.596717 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 715/1438] 127.0.0.1 () {54 vars in 901 bytes} [Sun Sep 25 04:29:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:29:47.603377 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3734912b-9b90-4356-8078-1b07b9a4702b None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:29:47.603808 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3734912b-9b90-4356-8078-1b07b9a4702b None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:29:47.603924 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3734912b-9b90-4356-8078-1b07b9a4702b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:29:47.678784 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-3734912b-9b90-4356-8078-1b07b9a4702b None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:29:47.679993 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-3734912b-9b90-4356-8078-1b07b9a4702b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:29:47.682026 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:29:47.775972 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 724/1439] 127.0.0.1 () {58 vars in 998 bytes} [Sun Sep 25 04:29:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:29:47.929124 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31b5c964-d25c-44ce-8b00-d6600da6e339 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:29:47.930670 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-31b5c964-d25c-44ce-8b00-d6600da6e339 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:29:48.076406 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31b5c964-d25c-44ce-8b00-d6600da6e339 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:29:48.103719 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:29:48.190416 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e924ff7f-11a1-437f-ac8b-51f6802902e2 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e703b42c53974daaa608b0ccc3f018b1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'ResellerAdmin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:29:48.191020 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e924ff7f-11a1-437f-ac8b-51f6802902e2 service nova] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:29:48.191303 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e924ff7f-11a1-437f-ac8b-51f6802902e2 service nova] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:29:48.191575 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e924ff7f-11a1-437f-ac8b-51f6802902e2 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:29:48.193221 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-e924ff7f-11a1-437f-ac8b-51f6802902e2 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:29:48.281838 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e924ff7f-11a1-437f-ac8b-51f6802902e2 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:29:48.283430 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e924ff7f-11a1-437f-ac8b-51f6802902e2 service nova] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:29:48.302074 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 716/1440] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:29:47 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 373 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 25 04:29:50.363826 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45112637-5c1b-4e59-8235-4d8315dc46d1 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:29:50.364395 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45112637-5c1b-4e59-8235-4d8315dc46d1 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:29:50.364395 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45112637-5c1b-4e59-8235-4d8315dc46d1 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:29:50.365772 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 725/1441] 127.0.0.1 () {54 vars in 917 bytes} [Sun Sep 25 04:29:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:29:50.371087 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b5e47c0-b13c-4d04-8a5f-7d6ca3dd985c None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:29:50.371482 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b5e47c0-b13c-4d04-8a5f-7d6ca3dd985c None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:29:50.371635 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b5e47c0-b13c-4d04-8a5f-7d6ca3dd985c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:29:50.412958 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:29:50.438050 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-7b5e47c0-b13c-4d04-8a5f-7d6ca3dd985c None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:29:50.439243 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-7b5e47c0-b13c-4d04-8a5f-7d6ca3dd985c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:29:50.537476 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 717/1442] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Sep 25 04:29:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:29:50.825802 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45112637-5c1b-4e59-8235-4d8315dc46d1 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:29:50.827514 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-45112637-5c1b-4e59-8235-4d8315dc46d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:29:50.937222 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45112637-5c1b-4e59-8235-4d8315dc46d1 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:29:50.972272 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:29:51.014340 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-701a815e-07e9-4b2c-8531-368169873cb1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:29:51.014945 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-701a815e-07e9-4b2c-8531-368169873cb1 service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:29:51.015219 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-701a815e-07e9-4b2c-8531-368169873cb1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:29:51.015485 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-701a815e-07e9-4b2c-8531-368169873cb1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:29:51.017108 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-701a815e-07e9-4b2c-8531-368169873cb1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:29:51.126378 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-701a815e-07e9-4b2c-8531-368169873cb1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:29:51.127724 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-701a815e-07e9-4b2c-8531-368169873cb1 service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:29:51.146347 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 726/1443] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:29:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 321 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:29:53.454541 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b06f84c-9988-476a-80cb-69e875d5f945 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:29:53.455111 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b06f84c-9988-476a-80cb-69e875d5f945 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:29:53.455111 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b06f84c-9988-476a-80cb-69e875d5f945 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:29:53.456700 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 718/1444] 127.0.0.1 () {54 vars in 917 bytes} [Sun Sep 25 04:29:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:29:53.462656 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1a5f624-d7b8-4dac-aeab-747d9e5614a6 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:29:53.463094 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1a5f624-d7b8-4dac-aeab-747d9e5614a6 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:29:53.463193 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1a5f624-d7b8-4dac-aeab-747d9e5614a6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:29:53.529380 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-c1a5f624-d7b8-4dac-aeab-747d9e5614a6 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:29:53.530597 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-c1a5f624-d7b8-4dac-aeab-747d9e5614a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:29:53.544017 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:29:53.627749 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 727/1445] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Sep 25 04:29:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:29:53.912560 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b06f84c-9988-476a-80cb-69e875d5f945 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:29:53.914132 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-7b06f84c-9988-476a-80cb-69e875d5f945 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:29:54.017436 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b06f84c-9988-476a-80cb-69e875d5f945 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:29:54.030614 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:29:54.091318 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b78f2fe-4ce3-4793-b8a1-edcf78c3aea6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:29:54.091922 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b78f2fe-4ce3-4793-b8a1-edcf78c3aea6 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:29:54.092201 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b78f2fe-4ce3-4793-b8a1-edcf78c3aea6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:29:54.092473 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b78f2fe-4ce3-4793-b8a1-edcf78c3aea6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:29:54.094096 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-1b78f2fe-4ce3-4793-b8a1-edcf78c3aea6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:29:54.204518 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b78f2fe-4ce3-4793-b8a1-edcf78c3aea6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:29:54.206157 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b78f2fe-4ce3-4793-b8a1-edcf78c3aea6 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:29:54.223979 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 719/1446] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:29:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 312 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:29:56.667125 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef7490e0-52e4-4130-ac36-3537277f4748 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:29:56.667701 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef7490e0-52e4-4130-ac36-3537277f4748 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:29:56.667701 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef7490e0-52e4-4130-ac36-3537277f4748 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:29:56.669050 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 728/1447] 127.0.0.1 () {54 vars in 917 bytes} [Sun Sep 25 04:29:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:29:56.674427 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d93ea045-377e-4917-a317-69c89482d26e None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:29:56.674836 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d93ea045-377e-4917-a317-69c89482d26e None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:29:56.674971 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d93ea045-377e-4917-a317-69c89482d26e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:29:56.723500 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:29:56.742106 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-d93ea045-377e-4917-a317-69c89482d26e None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:29:56.743314 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-d93ea045-377e-4917-a317-69c89482d26e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:29:56.839541 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 720/1448] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Sep 25 04:29:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:29:57.128110 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef7490e0-52e4-4130-ac36-3537277f4748 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:29:57.130452 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-ef7490e0-52e4-4130-ac36-3537277f4748 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:29:57.235627 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef7490e0-52e4-4130-ac36-3537277f4748 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:29:57.275435 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:29:57.310707 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40c01300-f55e-4a5c-8939-8a06b9c44b9d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:29:57.311310 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40c01300-f55e-4a5c-8939-8a06b9c44b9d service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:29:57.311584 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40c01300-f55e-4a5c-8939-8a06b9c44b9d service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:29:57.311852 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40c01300-f55e-4a5c-8939-8a06b9c44b9d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:29:57.313464 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-40c01300-f55e-4a5c-8939-8a06b9c44b9d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:29:57.420551 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40c01300-f55e-4a5c-8939-8a06b9c44b9d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:29:57.422076 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40c01300-f55e-4a5c-8939-8a06b9c44b9d service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:29:57.439659 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 729/1449] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:29:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 312 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:29:59.769682 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ed96e48-0cd8-4ede-b800-3368cbdbd618 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:29:59.770773 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ed96e48-0cd8-4ede-b800-3368cbdbd618 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:29:59.770773 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ed96e48-0cd8-4ede-b800-3368cbdbd618 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:29:59.772135 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 721/1450] 127.0.0.1 () {54 vars in 917 bytes} [Sun Sep 25 04:29:59 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:29:59.777640 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c294a9c3-44cd-46de-84e2-411a333dbdda None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:29:59.778064 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c294a9c3-44cd-46de-84e2-411a333dbdda None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:29:59.778202 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c294a9c3-44cd-46de-84e2-411a333dbdda None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:29:59.845914 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-c294a9c3-44cd-46de-84e2-411a333dbdda None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:29:59.847142 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-c294a9c3-44cd-46de-84e2-411a333dbdda None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:29:59.867675 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:29:59.943645 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 730/1451] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Sep 25 04:29:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:30:00.232317 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ed96e48-0cd8-4ede-b800-3368cbdbd618 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:30:00.233903 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-6ed96e48-0cd8-4ede-b800-3368cbdbd618 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:30:00.337554 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ed96e48-0cd8-4ede-b800-3368cbdbd618 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:30:00.357393 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:30:00.409558 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30a4e13e-7a84-4008-97f2-087629e8ff80 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:30:00.410154 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30a4e13e-7a84-4008-97f2-087629e8ff80 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:30:00.410432 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30a4e13e-7a84-4008-97f2-087629e8ff80 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:30:00.410700 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30a4e13e-7a84-4008-97f2-087629e8ff80 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:30:00.412337 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-30a4e13e-7a84-4008-97f2-087629e8ff80 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:30:00.521902 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30a4e13e-7a84-4008-97f2-087629e8ff80 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:30:00.523339 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30a4e13e-7a84-4008-97f2-087629e8ff80 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:30:00.541642 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 722/1452] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:30:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:30:02.737651 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39ce4887-fc7f-42c4-9527-c1889f32aa6d None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:30:02.738196 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39ce4887-fc7f-42c4-9527-c1889f32aa6d None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:30:02.738196 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39ce4887-fc7f-42c4-9527-c1889f32aa6d None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:30:02.740177 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 731/1453] 127.0.0.1 () {54 vars in 917 bytes} [Sun Sep 25 04:30:02 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:30:02.745643 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1a23902-500a-47a5-aa7e-704e89836126 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:30:02.746057 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1a23902-500a-47a5-aa7e-704e89836126 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:30:02.746225 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1a23902-500a-47a5-aa7e-704e89836126 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:30:02.809821 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:30:02.812534 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-b1a23902-500a-47a5-aa7e-704e89836126 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:30:02.814273 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-b1a23902-500a-47a5-aa7e-704e89836126 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:30:02.910949 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 723/1454] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Sep 25 04:30:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:30:03.193967 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39ce4887-fc7f-42c4-9527-c1889f32aa6d None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:30:03.195516 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-39ce4887-fc7f-42c4-9527-c1889f32aa6d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:30:03.298423 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39ce4887-fc7f-42c4-9527-c1889f32aa6d None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:30:03.346675 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:30:03.371792 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cefb34f-2b08-4276-b181-12eedfc312d6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:30:03.372393 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cefb34f-2b08-4276-b181-12eedfc312d6 service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:30:03.372690 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cefb34f-2b08-4276-b181-12eedfc312d6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:30:03.372958 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cefb34f-2b08-4276-b181-12eedfc312d6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:30:03.374528 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-0cefb34f-2b08-4276-b181-12eedfc312d6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:30:03.484423 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cefb34f-2b08-4276-b181-12eedfc312d6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:30:03.485895 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cefb34f-2b08-4276-b181-12eedfc312d6 service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:30:03.504004 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 732/1455] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:30:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:30:05.715476 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d891f7e-39b8-406f-ac19-c35782cabde1 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:30:05.716065 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d891f7e-39b8-406f-ac19-c35782cabde1 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:30:05.716065 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d891f7e-39b8-406f-ac19-c35782cabde1 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:30:05.717402 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 724/1456] 127.0.0.1 () {54 vars in 917 bytes} [Sun Sep 25 04:30:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:30:05.722770 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfe7e828-b49d-4b26-b0f1-9677dea2653c None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:30:05.723813 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfe7e828-b49d-4b26-b0f1-9677dea2653c None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:30:05.723903 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfe7e828-b49d-4b26-b0f1-9677dea2653c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:30:05.789350 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-dfe7e828-b49d-4b26-b0f1-9677dea2653c None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:30:05.790477 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-dfe7e828-b49d-4b26-b0f1-9677dea2653c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:30:05.822382 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:30:05.889980 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 733/1457] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Sep 25 04:30:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:30:06.180288 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d891f7e-39b8-406f-ac19-c35782cabde1 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:30:06.181903 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-9d891f7e-39b8-406f-ac19-c35782cabde1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:30:06.285155 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d891f7e-39b8-406f-ac19-c35782cabde1 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:30:06.314504 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:30:06.359538 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62fbba4f-abb7-4a3c-a6fc-6ccf51d6ade5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:30:06.360136 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62fbba4f-abb7-4a3c-a6fc-6ccf51d6ade5 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:30:06.360412 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62fbba4f-abb7-4a3c-a6fc-6ccf51d6ade5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:30:06.360700 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62fbba4f-abb7-4a3c-a6fc-6ccf51d6ade5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:30:06.362287 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-62fbba4f-abb7-4a3c-a6fc-6ccf51d6ade5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:30:06.470107 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62fbba4f-abb7-4a3c-a6fc-6ccf51d6ade5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:30:06.471542 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62fbba4f-abb7-4a3c-a6fc-6ccf51d6ade5 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:30:06.491141 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 725/1458] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:30:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:30:08.706448 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e668f13b-3971-471b-aff3-ff1170d39e97 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:30:08.707018 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e668f13b-3971-471b-aff3-ff1170d39e97 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:30:08.707120 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e668f13b-3971-471b-aff3-ff1170d39e97 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:30:08.709162 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 734/1459] 127.0.0.1 () {54 vars in 917 bytes} [Sun Sep 25 04:30:08 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:30:08.714669 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-695a7f9d-f5ab-4b34-8224-56b3d4064cf7 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:30:08.715092 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-695a7f9d-f5ab-4b34-8224-56b3d4064cf7 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:30:08.715192 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-695a7f9d-f5ab-4b34-8224-56b3d4064cf7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:30:08.782275 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-695a7f9d-f5ab-4b34-8224-56b3d4064cf7 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:30:08.783495 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-695a7f9d-f5ab-4b34-8224-56b3d4064cf7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:30:08.792308 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:30:08.881279 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 726/1460] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Sep 25 04:30:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:30:09.172357 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e668f13b-3971-471b-aff3-ff1170d39e97 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:30:09.173944 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-e668f13b-3971-471b-aff3-ff1170d39e97 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:30:09.276548 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e668f13b-3971-471b-aff3-ff1170d39e97 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:30:09.338215 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:30:09.356201 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-277e80f0-0c49-40a4-bed4-563b1f9c278a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:30:09.356837 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-277e80f0-0c49-40a4-bed4-563b1f9c278a service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:30:09.357122 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-277e80f0-0c49-40a4-bed4-563b1f9c278a service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:30:09.357394 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-277e80f0-0c49-40a4-bed4-563b1f9c278a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:30:09.359006 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-277e80f0-0c49-40a4-bed4-563b1f9c278a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:30:09.467254 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-277e80f0-0c49-40a4-bed4-563b1f9c278a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:30:09.468615 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-277e80f0-0c49-40a4-bed4-563b1f9c278a service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:30:09.486811 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 735/1461] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:30:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 315 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:30:11.702374 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-253b1a0c-5305-431d-92fb-e97adba8afe1 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:30:11.702978 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-253b1a0c-5305-431d-92fb-e97adba8afe1 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:30:11.702978 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-253b1a0c-5305-431d-92fb-e97adba8afe1 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:30:11.704261 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 727/1462] 127.0.0.1 () {54 vars in 917 bytes} [Sun Sep 25 04:30:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:30:11.710151 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f8cd597-2bad-4191-9326-2f8dabf0765b None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:30:11.710571 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f8cd597-2bad-4191-9326-2f8dabf0765b None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:30:11.710738 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f8cd597-2bad-4191-9326-2f8dabf0765b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:30:11.777003 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-0f8cd597-2bad-4191-9326-2f8dabf0765b None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:30:11.778118 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-0f8cd597-2bad-4191-9326-2f8dabf0765b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:30:11.816477 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:30:11.878504 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 736/1463] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Sep 25 04:30:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:30:12.161508 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-253b1a0c-5305-431d-92fb-e97adba8afe1 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:30:12.163078 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-253b1a0c-5305-431d-92fb-e97adba8afe1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:30:12.265076 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-253b1a0c-5305-431d-92fb-e97adba8afe1 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:30:12.303191 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:30:12.340786 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e53fe031-01e1-4c82-9afd-2c1aaacd2df2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:30:12.341393 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e53fe031-01e1-4c82-9afd-2c1aaacd2df2 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:30:12.341673 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e53fe031-01e1-4c82-9afd-2c1aaacd2df2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:30:12.341946 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e53fe031-01e1-4c82-9afd-2c1aaacd2df2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:30:12.343535 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-e53fe031-01e1-4c82-9afd-2c1aaacd2df2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:30:12.450876 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e53fe031-01e1-4c82-9afd-2c1aaacd2df2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:30:12.452220 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e53fe031-01e1-4c82-9afd-2c1aaacd2df2 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:30:12.473933 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 728/1464] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:30:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 313 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:30:14.672421 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-331ad465-1860-4cc7-9448-037badbdcdb8 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:30:14.673016 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-331ad465-1860-4cc7-9448-037badbdcdb8 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:30:14.673016 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-331ad465-1860-4cc7-9448-037badbdcdb8 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:30:14.674346 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 737/1465] 127.0.0.1 () {54 vars in 917 bytes} [Sun Sep 25 04:30:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:30:14.679929 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06c78d27-5e8e-4522-9872-1a570c9f72f5 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:30:14.680362 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06c78d27-5e8e-4522-9872-1a570c9f72f5 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:30:14.680468 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06c78d27-5e8e-4522-9872-1a570c9f72f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:30:14.746803 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-06c78d27-5e8e-4522-9872-1a570c9f72f5 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:30:14.747952 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-06c78d27-5e8e-4522-9872-1a570c9f72f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:30:14.765962 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:30:14.845482 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 729/1466] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Sep 25 04:30:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:30:15.125753 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-331ad465-1860-4cc7-9448-037badbdcdb8 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:30:15.127306 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-331ad465-1860-4cc7-9448-037badbdcdb8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:30:15.232402 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-331ad465-1860-4cc7-9448-037badbdcdb8 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:30:15.299034 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:30:15.306614 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0260a93-8ec9-4075-af09-badad967e499 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:30:15.307214 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0260a93-8ec9-4075-af09-badad967e499 service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:30:15.307489 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0260a93-8ec9-4075-af09-badad967e499 service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:30:15.307756 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0260a93-8ec9-4075-af09-badad967e499 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:30:15.309386 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-e0260a93-8ec9-4075-af09-badad967e499 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:30:15.418285 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0260a93-8ec9-4075-af09-badad967e499 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:30:15.419711 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0260a93-8ec9-4075-af09-badad967e499 service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:30:15.437342 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 738/1467] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:30:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 312 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:30:17.612620 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-862acc53-17f3-4d66-a620-5954fa714e8a None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:30:17.613196 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-862acc53-17f3-4d66-a620-5954fa714e8a None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:30:17.613196 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-862acc53-17f3-4d66-a620-5954fa714e8a None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:30:17.614620 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 730/1468] 127.0.0.1 () {54 vars in 917 bytes} [Sun Sep 25 04:30:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:30:17.620342 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-064c23bf-cabd-4f16-8787-c125a322a7a1 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:30:17.620832 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-064c23bf-cabd-4f16-8787-c125a322a7a1 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:30:17.620936 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-064c23bf-cabd-4f16-8787-c125a322a7a1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:30:17.686641 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-064c23bf-cabd-4f16-8787-c125a322a7a1 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:30:17.687831 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-064c23bf-cabd-4f16-8787-c125a322a7a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:30:17.737139 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:30:17.786526 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 739/1469] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Sep 25 04:30:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:30:18.070845 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-862acc53-17f3-4d66-a620-5954fa714e8a None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:30:18.072443 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-862acc53-17f3-4d66-a620-5954fa714e8a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:30:18.178315 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-862acc53-17f3-4d66-a620-5954fa714e8a None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:30:18.226710 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:30:18.252355 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b7ad4d9-0331-48ff-a6e4-7609d02a6ac4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:30:18.252998 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b7ad4d9-0331-48ff-a6e4-7609d02a6ac4 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:30:18.253281 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b7ad4d9-0331-48ff-a6e4-7609d02a6ac4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:30:18.253555 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b7ad4d9-0331-48ff-a6e4-7609d02a6ac4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:30:18.255231 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-6b7ad4d9-0331-48ff-a6e4-7609d02a6ac4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:30:18.369396 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b7ad4d9-0331-48ff-a6e4-7609d02a6ac4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:30:18.371781 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b7ad4d9-0331-48ff-a6e4-7609d02a6ac4 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:30:18.389140 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 731/1470] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:30:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 319 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:30:27.784462 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=22 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:30:28.386847 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=66 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:30:38.607546 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cc74d2b-f38d-4014-8682-b5fa83510b14 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:30:38.608087 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cc74d2b-f38d-4014-8682-b5fa83510b14 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:30:38.608087 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cc74d2b-f38d-4014-8682-b5fa83510b14 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:30:38.609457 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 740/1471] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:30:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:30:38.616014 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f99276-daa2-4b29-8936-56fd3bc1dd37 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:30:38.616480 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f99276-daa2-4b29-8936-56fd3bc1dd37 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:30:38.616581 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f99276-daa2-4b29-8936-56fd3bc1dd37 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:30:38.686133 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-f9f99276-daa2-4b29-8936-56fd3bc1dd37 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:30:38.687356 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-f9f99276-daa2-4b29-8936-56fd3bc1dd37 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:30:38.790730 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 732/1472] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:30:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:30:39.097389 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-239f358f-c8e0-40fc-b955-d3fb93160dbf None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:30:39.098092 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-239f358f-c8e0-40fc-b955-d3fb93160dbf None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:30:39.098092 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-239f358f-c8e0-40fc-b955-d3fb93160dbf None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:30:39.099431 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 741/1473] 10.222.0.25 () {54 vars in 962 bytes} [Sun Sep 25 04:30:39 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:30:39.105233 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83fcb2b8-adb9-41dc-8a8f-30caa1c15717 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:30:39.105525 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83fcb2b8-adb9-41dc-8a8f-30caa1c15717 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:30:39.105802 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83fcb2b8-adb9-41dc-8a8f-30caa1c15717 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:30:39.186868 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-83fcb2b8-adb9-41dc-8a8f-30caa1c15717 None None] MFA Rules not processed for user `316eaa67d5464091af47efd489b96b87`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:30:39.188078 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-83fcb2b8-adb9-41dc-8a8f-30caa1c15717 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:30:39.210380 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:30:39.270204 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 733/1474] 10.222.0.25 () {58 vars in 1059 bytes} [Sun Sep 25 04:30:39 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:30:39.278129 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-239f358f-c8e0-40fc-b955-d3fb93160dbf None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:30:39.279637 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-239f358f-c8e0-40fc-b955-d3fb93160dbf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:30:39.390817 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-239f358f-c8e0-40fc-b955-d3fb93160dbf None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:30:39.473937 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95555a4e-d13c-4595-bda9-bc1f7ee622cc service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '316eaa67d5464091af47efd489b96b87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:30:39.474558 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95555a4e-d13c-4595-bda9-bc1f7ee622cc service glance] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:30:39.474835 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95555a4e-d13c-4595-bda9-bc1f7ee622cc service glance] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:30:39.475234 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95555a4e-d13c-4595-bda9-bc1f7ee622cc service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:30:39.476969 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-95555a4e-d13c-4595-bda9-bc1f7ee622cc service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:30:39.535212 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:30:39.593652 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95555a4e-d13c-4595-bda9-bc1f7ee622cc service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:30:39.595020 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95555a4e-d13c-4595-bda9-bc1f7ee622cc service glance] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:30:39.615424 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 742/1475] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:30:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 338 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:30:41.888061 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b05b0dff-e630-49ac-9fd6-7a0c46ea02df None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:30:41.888588 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b05b0dff-e630-49ac-9fd6-7a0c46ea02df None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:30:41.888588 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b05b0dff-e630-49ac-9fd6-7a0c46ea02df None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:30:41.890037 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 734/1476] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:30:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:30:41.895740 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37d09d50-f522-48ed-a8aa-c6a314112184 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:30:41.896166 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37d09d50-f522-48ed-a8aa-c6a314112184 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:30:41.896272 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37d09d50-f522-48ed-a8aa-c6a314112184 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:30:41.963103 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-37d09d50-f522-48ed-a8aa-c6a314112184 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:30:41.964313 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-37d09d50-f522-48ed-a8aa-c6a314112184 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:30:42.061039 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 743/1477] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:30:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:30:42.338275 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b05b0dff-e630-49ac-9fd6-7a0c46ea02df None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:30:42.339907 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-b05b0dff-e630-49ac-9fd6-7a0c46ea02df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:30:42.474776 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b05b0dff-e630-49ac-9fd6-7a0c46ea02df None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:30:42.524840 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:30:42.569097 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c2b514d-ffda-4129-8dd3-44df95a4a1dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:30:42.569701 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c2b514d-ffda-4129-8dd3-44df95a4a1dd admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:30:42.569980 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c2b514d-ffda-4129-8dd3-44df95a4a1dd admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:30:42.570253 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c2b514d-ffda-4129-8dd3-44df95a4a1dd admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:30:42.570880 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c2b514d-ffda-4129-8dd3-44df95a4a1dd admin admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:30:42.572665 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: WARNING py.warnings [None req-8c2b514d-ffda-4129-8dd3-44df95a4a1dd admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_projects": "(role:reader and system_scope:all) or (role:reader and domain_id:%(target.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 25 04:30:42.572665 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: warnings.warn(msg) Sep 25 04:30:42.572665 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: Sep 25 04:30:42.573822 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c2b514d-ffda-4129-8dd3-44df95a4a1dd admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:30:42.593354 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 735/1478] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Sep 25 04:30:42 2022] GET /identity/v3/projects => generated 2553 bytes in 256 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:30:44.787413 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fad2db8-5116-4740-a1a2-d8369aa55988 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:30:44.787959 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fad2db8-5116-4740-a1a2-d8369aa55988 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:30:44.787959 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fad2db8-5116-4740-a1a2-d8369aa55988 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:30:44.789244 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 744/1479] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:30:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:30:44.794706 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8464cc12-b997-4dd4-97d1-1af855b0ea6a None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:30:44.795114 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8464cc12-b997-4dd4-97d1-1af855b0ea6a None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:30:44.795262 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8464cc12-b997-4dd4-97d1-1af855b0ea6a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:30:44.860973 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-8464cc12-b997-4dd4-97d1-1af855b0ea6a None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:30:44.862252 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-8464cc12-b997-4dd4-97d1-1af855b0ea6a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:30:44.959425 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 736/1480] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:30:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:30:45.258480 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fad2db8-5116-4740-a1a2-d8369aa55988 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:30:45.260074 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-1fad2db8-5116-4740-a1a2-d8369aa55988 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:30:45.270663 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:30:45.405855 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fad2db8-5116-4740-a1a2-d8369aa55988 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:30:45.485243 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:30:45.518314 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83d31559-1574-4aa7-a0f8-695b6274c91e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e703b42c53974daaa608b0ccc3f018b1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'ResellerAdmin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:30:45.519362 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83d31559-1574-4aa7-a0f8-695b6274c91e service nova] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:30:45.519647 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83d31559-1574-4aa7-a0f8-695b6274c91e service nova] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:30:45.519899 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83d31559-1574-4aa7-a0f8-695b6274c91e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:30:45.521531 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-83d31559-1574-4aa7-a0f8-695b6274c91e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:30:45.611441 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83d31559-1574-4aa7-a0f8-695b6274c91e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:30:45.613807 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83d31559-1574-4aa7-a0f8-695b6274c91e service nova] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:30:45.631498 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 745/1481] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:30:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 373 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:30:47.997990 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b9116c-aea7-4c55-84c5-0a6fbb4755b9 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:30:47.998570 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b9116c-aea7-4c55-84c5-0a6fbb4755b9 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:30:47.998570 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b9116c-aea7-4c55-84c5-0a6fbb4755b9 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:30:47.999960 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 737/1482] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:30:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:30:48.005906 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eba4de7a-96ea-4e4f-a34b-9246314ea109 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:30:48.006345 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eba4de7a-96ea-4e4f-a34b-9246314ea109 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:30:48.006458 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eba4de7a-96ea-4e4f-a34b-9246314ea109 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:30:48.072355 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-eba4de7a-96ea-4e4f-a34b-9246314ea109 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:30:48.073601 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-eba4de7a-96ea-4e4f-a34b-9246314ea109 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:30:48.119291 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:30:48.169666 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 746/1483] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:30:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:30:48.462418 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8b9116c-aea7-4c55-84c5-0a6fbb4755b9 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:30:48.464003 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-d8b9116c-aea7-4c55-84c5-0a6fbb4755b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:30:48.488612 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:30:48.568681 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8b9116c-aea7-4c55-84c5-0a6fbb4755b9 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:30:48.643038 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc2bc100-b3bf-4ed6-a777-b21e6112805c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:30:48.643655 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc2bc100-b3bf-4ed6-a777-b21e6112805c service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:30:48.643928 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc2bc100-b3bf-4ed6-a777-b21e6112805c service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:30:48.644199 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc2bc100-b3bf-4ed6-a777-b21e6112805c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:30:48.645899 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-bc2bc100-b3bf-4ed6-a777-b21e6112805c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:30:48.718987 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:30:48.754762 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc2bc100-b3bf-4ed6-a777-b21e6112805c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:30:48.756190 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc2bc100-b3bf-4ed6-a777-b21e6112805c service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:30:48.774478 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 738/1484] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:30:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 313 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:30:50.986440 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35c418e4-49b2-4c6a-a0e0-4dcd473279ac None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:30:50.987551 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35c418e4-49b2-4c6a-a0e0-4dcd473279ac None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:30:50.987551 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35c418e4-49b2-4c6a-a0e0-4dcd473279ac None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:30:50.988959 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 747/1485] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:30:50 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:30:50.994406 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-476062db-ec2a-41fd-8ecb-422b4df77635 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:30:50.994863 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-476062db-ec2a-41fd-8ecb-422b4df77635 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:30:50.994979 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-476062db-ec2a-41fd-8ecb-422b4df77635 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:30:51.060574 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-476062db-ec2a-41fd-8ecb-422b4df77635 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:30:51.061817 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-476062db-ec2a-41fd-8ecb-422b4df77635 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:30:51.158679 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 739/1486] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:30:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 04:30:51.448591 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35c418e4-49b2-4c6a-a0e0-4dcd473279ac None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:30:51.450171 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-35c418e4-49b2-4c6a-a0e0-4dcd473279ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:30:51.556610 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35c418e4-49b2-4c6a-a0e0-4dcd473279ac None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:30:51.628806 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:30:51.630349 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ba72762-24e6-4c40-aa2b-2551539217ea service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:30:51.631082 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ba72762-24e6-4c40-aa2b-2551539217ea service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:30:51.631556 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ba72762-24e6-4c40-aa2b-2551539217ea service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:30:51.631991 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ba72762-24e6-4c40-aa2b-2551539217ea service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:30:51.633845 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-6ba72762-24e6-4c40-aa2b-2551539217ea service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:30:51.745563 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ba72762-24e6-4c40-aa2b-2551539217ea service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:30:51.746978 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ba72762-24e6-4c40-aa2b-2551539217ea service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:30:51.765222 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 748/1487] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:30:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 317 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:30:54.027724 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05a7b428-1dcb-4a51-9cf0-6284b7f1f3c5 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:30:54.028295 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05a7b428-1dcb-4a51-9cf0-6284b7f1f3c5 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:30:54.028295 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05a7b428-1dcb-4a51-9cf0-6284b7f1f3c5 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:30:54.029667 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 740/1488] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:30:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:30:54.035816 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e1ee264-1f90-4be8-af78-56139414cad6 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:30:54.036236 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e1ee264-1f90-4be8-af78-56139414cad6 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:30:54.036369 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e1ee264-1f90-4be8-af78-56139414cad6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:30:54.102861 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-0e1ee264-1f90-4be8-af78-56139414cad6 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:30:54.104063 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-0e1ee264-1f90-4be8-af78-56139414cad6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:30:54.156750 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:30:54.200552 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 749/1489] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:30:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:30:54.495577 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05a7b428-1dcb-4a51-9cf0-6284b7f1f3c5 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:30:54.497088 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-05a7b428-1dcb-4a51-9cf0-6284b7f1f3c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:30:54.532887 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:30:54.600373 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05a7b428-1dcb-4a51-9cf0-6284b7f1f3c5 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:30:54.672954 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e38c30e1-e95c-4310-b319-9a57e1abf143 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:30:54.673569 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e38c30e1-e95c-4310-b319-9a57e1abf143 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:30:54.673848 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e38c30e1-e95c-4310-b319-9a57e1abf143 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:30:54.674121 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e38c30e1-e95c-4310-b319-9a57e1abf143 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:30:54.676259 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-e38c30e1-e95c-4310-b319-9a57e1abf143 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:30:54.756579 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:30:54.785924 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e38c30e1-e95c-4310-b319-9a57e1abf143 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:30:54.787359 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e38c30e1-e95c-4310-b319-9a57e1abf143 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:30:54.806097 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 741/1490] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:30:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:30:57.246827 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b096f99-9134-4d20-b346-d2f21cbae2d0 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:30:57.247379 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b096f99-9134-4d20-b346-d2f21cbae2d0 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:30:57.247379 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b096f99-9134-4d20-b346-d2f21cbae2d0 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:30:57.248773 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 750/1491] 10.222.0.25 () {54 vars in 927 bytes} [Sun Sep 25 04:30:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:30:57.254099 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e478c80-fac7-4d72-97be-55433c1cfa73 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:30:57.254563 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e478c80-fac7-4d72-97be-55433c1cfa73 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:30:57.254672 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e478c80-fac7-4d72-97be-55433c1cfa73 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:30:57.321262 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-6e478c80-fac7-4d72-97be-55433c1cfa73 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:30:57.322456 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-6e478c80-fac7-4d72-97be-55433c1cfa73 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:30:57.417131 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 742/1492] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Sep 25 04:30:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:30:57.710903 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b096f99-9134-4d20-b346-d2f21cbae2d0 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:30:57.712473 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-9b096f99-9134-4d20-b346-d2f21cbae2d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:30:57.814381 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b096f99-9134-4d20-b346-d2f21cbae2d0 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:30:57.889235 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73c96bfb-a260-484d-b5c4-a6c6e08c1d2b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:30:57.889845 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73c96bfb-a260-484d-b5c4-a6c6e08c1d2b service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:30:57.890130 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73c96bfb-a260-484d-b5c4-a6c6e08c1d2b service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:30:57.890400 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73c96bfb-a260-484d-b5c4-a6c6e08c1d2b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:30:57.892001 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-73c96bfb-a260-484d-b5c4-a6c6e08c1d2b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:30:57.901139 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:30:58.002142 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73c96bfb-a260-484d-b5c4-a6c6e08c1d2b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:30:58.003616 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73c96bfb-a260-484d-b5c4-a6c6e08c1d2b service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:30:58.021903 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 751/1493] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:30:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 312 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:31:07.415600 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=82 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:31:08.019645 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=50 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:32:11.279413 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6d49dfe-e7c3-4239-a848-48057a936af6 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:32:11.280035 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6d49dfe-e7c3-4239-a848-48057a936af6 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:32:11.280035 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6d49dfe-e7c3-4239-a848-48057a936af6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:32:11.356171 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-b6d49dfe-e7c3-4239-a848-48057a936af6 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:32:11.357323 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-b6d49dfe-e7c3-4239-a848-48057a936af6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:32:11.450893 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 743/1494] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:32:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:32:11.460028 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccbc17fb-96c0-420f-a2eb-5f335a70de01 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:32:11.460603 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccbc17fb-96c0-420f-a2eb-5f335a70de01 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:32:11.460741 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccbc17fb-96c0-420f-a2eb-5f335a70de01 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:32:11.536845 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-ccbc17fb-96c0-420f-a2eb-5f335a70de01 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:32:11.538051 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-ccbc17fb-96c0-420f-a2eb-5f335a70de01 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:32:11.633579 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 752/1495] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:32:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:32:11.638063 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6d49dfe-e7c3-4239-a848-48057a936af6 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:32:11.639273 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-b6d49dfe-e7c3-4239-a848-48057a936af6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:32:11.651552 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6d49dfe-e7c3-4239-a848-48057a936af6 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:32:11.716407 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:32:11.749264 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac358aca-f7eb-4ff4-934c-43f27603e08d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:32:11.749895 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac358aca-f7eb-4ff4-934c-43f27603e08d admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:32:11.750191 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac358aca-f7eb-4ff4-934c-43f27603e08d admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:32:11.750447 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac358aca-f7eb-4ff4-934c-43f27603e08d admin admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:32:11.751082 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac358aca-f7eb-4ff4-934c-43f27603e08d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:32:11.751383 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac358aca-f7eb-4ff4-934c-43f27603e08d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:32:11.752392 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: WARNING py.warnings [None req-ac358aca-f7eb-4ff4-934c-43f27603e08d admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Sep 25 04:32:11.752392 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: warnings.warn(msg) Sep 25 04:32:11.752392 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: Sep 25 04:32:11.753307 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac358aca-f7eb-4ff4-934c-43f27603e08d admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:32:11.764579 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 744/1496] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Sep 25 04:32:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 127 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:32:11.773558 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccbc17fb-96c0-420f-a2eb-5f335a70de01 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:32:11.783483 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccbc17fb-96c0-420f-a2eb-5f335a70de01 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:32:11.845193 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:32:11.879037 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8397906-3083-4657-b9ac-61eb318f9112 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:32:11.879647 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8397906-3083-4657-b9ac-61eb318f9112 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:32:11.879927 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8397906-3083-4657-b9ac-61eb318f9112 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:32:11.880198 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8397906-3083-4657-b9ac-61eb318f9112 admin admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:32:11.880855 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8397906-3083-4657-b9ac-61eb318f9112 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:32:11.882577 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: WARNING py.warnings [None req-f8397906-3083-4657-b9ac-61eb318f9112 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 25 04:32:11.882577 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: warnings.warn(msg) Sep 25 04:32:11.882577 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: Sep 25 04:32:11.883865 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8397906-3083-4657-b9ac-61eb318f9112 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:32:11.940330 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 753/1497] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:32:11 2022] POST /identity/v3/projects => generated 377 bytes in 167 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:32:11.944940 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac358aca-f7eb-4ff4-934c-43f27603e08d admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:32:11.946459 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-ac358aca-f7eb-4ff4-934c-43f27603e08d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:32:11.980436 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac358aca-f7eb-4ff4-934c-43f27603e08d admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:32:12.073673 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4be4349-dc8d-4163-aa3d-0bbb87943b7b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:32:12.074272 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4be4349-dc8d-4163-aa3d-0bbb87943b7b admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:32:12.074552 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4be4349-dc8d-4163-aa3d-0bbb87943b7b admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:32:12.074821 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4be4349-dc8d-4163-aa3d-0bbb87943b7b admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:32:12.075448 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4be4349-dc8d-4163-aa3d-0bbb87943b7b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:32:12.076443 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: WARNING py.warnings [None req-b4be4349-dc8d-4163-aa3d-0bbb87943b7b admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 25 04:32:12.076443 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: warnings.warn(msg) Sep 25 04:32:12.076443 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: Sep 25 04:32:12.077665 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4be4349-dc8d-4163-aa3d-0bbb87943b7b admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:32:12.119585 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 745/1498] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:32:11 2022] POST /identity/v3/users => generated 289 bytes in 176 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:32:12.123606 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8397906-3083-4657-b9ac-61eb318f9112 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:32:12.126030 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-f8397906-3083-4657-b9ac-61eb318f9112 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:32:12.187585 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8397906-3083-4657-b9ac-61eb318f9112 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:32:12.270769 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:32:12.282336 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36bb91ac-a99a-4cde-aa5d-c26987229cbc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:32:12.282941 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36bb91ac-a99a-4cde-aa5d-c26987229cbc admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:32:12.283217 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36bb91ac-a99a-4cde-aa5d-c26987229cbc admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:32:12.283482 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36bb91ac-a99a-4cde-aa5d-c26987229cbc admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:32:12.284117 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36bb91ac-a99a-4cde-aa5d-c26987229cbc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:32:12.285150 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: WARNING py.warnings [None req-36bb91ac-a99a-4cde-aa5d-c26987229cbc admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Sep 25 04:32:12.285150 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: warnings.warn(msg) Sep 25 04:32:12.285150 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: Sep 25 04:32:12.286039 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36bb91ac-a99a-4cde-aa5d-c26987229cbc admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:32:12.298601 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 754/1499] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:32:12 2022] GET /identity/v3/roles => generated 1420 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:32:12.302667 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4be4349-dc8d-4163-aa3d-0bbb87943b7b admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:32:12.304193 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-b4be4349-dc8d-4163-aa3d-0bbb87943b7b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:32:12.315999 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4be4349-dc8d-4163-aa3d-0bbb87943b7b admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:32:12.341306 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:32:12.412310 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-435cf439-5846-4ff9-8d0f-52b1d49745eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:32:12.412952 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-435cf439-5846-4ff9-8d0f-52b1d49745eb admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:32:12.413235 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-435cf439-5846-4ff9-8d0f-52b1d49745eb admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:32:12.413507 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-435cf439-5846-4ff9-8d0f-52b1d49745eb admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:32:12.414146 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-435cf439-5846-4ff9-8d0f-52b1d49745eb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:32:12.415161 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: WARNING py.warnings [None req-435cf439-5846-4ff9-8d0f-52b1d49745eb admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Sep 25 04:32:12.415161 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: warnings.warn(msg) Sep 25 04:32:12.415161 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: Sep 25 04:32:12.416046 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-435cf439-5846-4ff9-8d0f-52b1d49745eb admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:32:12.428281 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 746/1500] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:32:12 2022] GET /identity/v3/roles => generated 1420 bytes in 126 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:32:12.432528 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36bb91ac-a99a-4cde-aa5d-c26987229cbc admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:32:12.434082 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-36bb91ac-a99a-4cde-aa5d-c26987229cbc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:32:12.468170 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36bb91ac-a99a-4cde-aa5d-c26987229cbc admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:32:12.562451 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfc272f0-cc4a-4337-a5b1-d6621416ca8a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:32:12.563096 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfc272f0-cc4a-4337-a5b1-d6621416ca8a admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:32:12.563371 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfc272f0-cc4a-4337-a5b1-d6621416ca8a admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:32:12.563641 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfc272f0-cc4a-4337-a5b1-d6621416ca8a admin admin] PATH_INFO: `/v3/projects/30c1f31b50ce482290cd14d309e0b78f/users/79c4aa36674c47bfbc494b4b1f5926ac/roles/8f6cc574056c4278bc9778f14dc4b411` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:32:12.585552 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfc272f0-cc4a-4337-a5b1-d6621416ca8a admin admin] RBAC: Authorizing `identity:create_grant(project_id=30c1f31b50ce482290cd14d309e0b78f, user_id=79c4aa36674c47bfbc494b4b1f5926ac, role_id=8f6cc574056c4278bc9778f14dc4b411)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:32:12.586563 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: WARNING py.warnings [None req-dfc272f0-cc4a-4337-a5b1-d6621416ca8a admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 25 04:32:12.586563 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: warnings.warn(msg) Sep 25 04:32:12.586563 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: Sep 25 04:32:12.588479 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfc272f0-cc4a-4337-a5b1-d6621416ca8a admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:32:12.613593 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 755/1501] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:32:12 2022] PUT /identity/v3/projects/30c1f31b50ce482290cd14d309e0b78f/users/79c4aa36674c47bfbc494b4b1f5926ac/roles/8f6cc574056c4278bc9778f14dc4b411 => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:32:12.626267 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b48d8549-a1ef-455f-a2eb-5800285f4f59 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:32:12.626607 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b48d8549-a1ef-455f-a2eb-5800285f4f59 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:32:12.626887 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b48d8549-a1ef-455f-a2eb-5800285f4f59 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:32:12.684306 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-b48d8549-a1ef-455f-a2eb-5800285f4f59 None None] MFA Rules not processed for user `79c4aa36674c47bfbc494b4b1f5926ac`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:32:12.685541 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-b48d8549-a1ef-455f-a2eb-5800285f4f59 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:32:12.762919 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 747/1502] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:32:12 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 04:32:12.776056 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a38a0d-5a0e-457b-ae68-386390e93891 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:32:12.776330 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a38a0d-5a0e-457b-ae68-386390e93891 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:32:12.776603 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a38a0d-5a0e-457b-ae68-386390e93891 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:32:12.777950 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 756/1503] 10.222.0.25 () {54 vars in 963 bytes} [Sun Sep 25 04:32:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:32:12.784267 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c5f40ce-4f48-4e6f-b5cb-6781df5b777d None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:32:12.784561 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c5f40ce-4f48-4e6f-b5cb-6781df5b777d None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:32:12.784868 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c5f40ce-4f48-4e6f-b5cb-6781df5b777d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:32:12.819743 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:32:12.865478 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-0c5f40ce-4f48-4e6f-b5cb-6781df5b777d None None] MFA Rules not processed for user `ffbc136c77b84e8c87128f7247f4ec03`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:32:12.866696 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-0c5f40ce-4f48-4e6f-b5cb-6781df5b777d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:32:12.942495 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 748/1504] 10.222.0.25 () {58 vars in 1060 bytes} [Sun Sep 25 04:32:12 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:32:12.950198 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45a38a0d-5a0e-457b-ae68-386390e93891 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:32:12.951734 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-45a38a0d-5a0e-457b-ae68-386390e93891 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:32:13.002536 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:32:13.055839 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45a38a0d-5a0e-457b-ae68-386390e93891 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:32:13.131076 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a485eff7-49be-4ee7-a256-b6226b3180f1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ffbc136c77b84e8c87128f7247f4ec03', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:32:13.131679 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a485eff7-49be-4ee7-a256-b6226b3180f1 service cinder] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:32:13.131957 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a485eff7-49be-4ee7-a256-b6226b3180f1 service cinder] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:32:13.132226 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a485eff7-49be-4ee7-a256-b6226b3180f1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:32:13.133859 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-a485eff7-49be-4ee7-a256-b6226b3180f1 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:32:13.232124 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a485eff7-49be-4ee7-a256-b6226b3180f1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:32:13.233581 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a485eff7-49be-4ee7-a256-b6226b3180f1 service cinder] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:32:13.237717 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:32:13.252890 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 757/1505] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:32:12 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 303 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:32:13.298253 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f96254c-ccfb-46bc-b7c8-3da13e7fc3cd None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:32:13.298545 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f96254c-ccfb-46bc-b7c8-3da13e7fc3cd None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:32:13.298823 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f96254c-ccfb-46bc-b7c8-3da13e7fc3cd None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:32:13.300077 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 749/1506] 10.222.0.25 () {52 vars in 830 bytes} [Sun Sep 25 04:32:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:32:13.303628 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a485eff7-49be-4ee7-a256-b6226b3180f1 service cinder] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:32:13.304849 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-a485eff7-49be-4ee7-a256-b6226b3180f1 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:32:13.402316 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a485eff7-49be-4ee7-a256-b6226b3180f1 service cinder] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:32:13.497121 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-982d32e8-f363-4da9-8b56-a25adddc004a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:32:13.497762 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-982d32e8-f363-4da9-8b56-a25adddc004a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:32:13.498013 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-982d32e8-f363-4da9-8b56-a25adddc004a admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:32:13.498281 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-982d32e8-f363-4da9-8b56-a25adddc004a admin admin] PATH_INFO: `/v3/users/79c4aa36674c47bfbc494b4b1f5926ac` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:32:13.511937 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-982d32e8-f363-4da9-8b56-a25adddc004a admin admin] RBAC: Authorizing `identity:delete_user(user_id=79c4aa36674c47bfbc494b4b1f5926ac)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:32:13.512984 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: WARNING py.warnings [None req-982d32e8-f363-4da9-8b56-a25adddc004a admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 25 04:32:13.512984 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: warnings.warn(msg) Sep 25 04:32:13.512984 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: Sep 25 04:32:13.514188 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-982d32e8-f363-4da9-8b56-a25adddc004a admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:32:13.524381 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:32:13.580882 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-982d32e8-f363-4da9-8b56-a25adddc004a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '79c4aa36674c47bfbc494b4b1f5926ac'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:32:13.589855 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-982d32e8-f363-4da9-8b56-a25adddc004a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '79c4aa36674c47bfbc494b4b1f5926ac'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:32:13.615691 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-982d32e8-f363-4da9-8b56-a25adddc004a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '79c4aa36674c47bfbc494b4b1f5926ac'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:32:13.616148 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-982d32e8-f363-4da9-8b56-a25adddc004a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '79c4aa36674c47bfbc494b4b1f5926ac'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:32:13.630047 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-982d32e8-f363-4da9-8b56-a25adddc004a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '79c4aa36674c47bfbc494b4b1f5926ac'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:32:13.635884 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 758/1507] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:32:13 2022] DELETE /identity/v3/users/79c4aa36674c47bfbc494b4b1f5926ac => generated 0 bytes in 333 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:32:13.641000 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5743ebc3-b7d6-448d-8e1c-5cb0696fb69f None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:32:13.641483 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5743ebc3-b7d6-448d-8e1c-5cb0696fb69f None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:32:13.641591 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5743ebc3-b7d6-448d-8e1c-5cb0696fb69f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:32:13.698853 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-5743ebc3-b7d6-448d-8e1c-5cb0696fb69f None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:32:13.699755 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-5743ebc3-b7d6-448d-8e1c-5cb0696fb69f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:32:13.774720 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:32:13.787562 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 750/1508] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:32:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:32:13.797763 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-982d32e8-f363-4da9-8b56-a25adddc004a admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:32:13.798978 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-982d32e8-f363-4da9-8b56-a25adddc004a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:32:13.898718 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-982d32e8-f363-4da9-8b56-a25adddc004a admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:32:13.953482 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:DELETE=10,keystone:SELECT=89,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:32:13.975500 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ca88604-e004-4ccc-80f2-a01a16cfe476 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:32:13.976122 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca88604-e004-4ccc-80f2-a01a16cfe476 service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:32:13.976402 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca88604-e004-4ccc-80f2-a01a16cfe476 service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:32:13.976696 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca88604-e004-4ccc-80f2-a01a16cfe476 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:32:13.978317 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-7ca88604-e004-4ccc-80f2-a01a16cfe476 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:32:14.087333 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ca88604-e004-4ccc-80f2-a01a16cfe476 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:32:14.088892 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ca88604-e004-4ccc-80f2-a01a16cfe476 service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:32:14.107425 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 759/1509] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:32:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:32:14.440950 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5743ebc3-b7d6-448d-8e1c-5cb0696fb69f None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:32:14.442328 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-5743ebc3-b7d6-448d-8e1c-5cb0696fb69f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:32:14.457008 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5743ebc3-b7d6-448d-8e1c-5cb0696fb69f None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:32:14.558699 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60b64017-9714-4d4f-923f-737ce5be5fa7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:32:14.559336 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60b64017-9714-4d4f-923f-737ce5be5fa7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:32:14.559615 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60b64017-9714-4d4f-923f-737ce5be5fa7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:32:14.559884 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60b64017-9714-4d4f-923f-737ce5be5fa7 admin admin] PATH_INFO: `/v3/projects/30c1f31b50ce482290cd14d309e0b78f` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:32:14.570420 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60b64017-9714-4d4f-923f-737ce5be5fa7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=30c1f31b50ce482290cd14d309e0b78f)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:32:14.571433 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: WARNING py.warnings [None req-60b64017-9714-4d4f-923f-737ce5be5fa7 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 25 04:32:14.571433 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: warnings.warn(msg) Sep 25 04:32:14.571433 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: Sep 25 04:32:14.572630 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60b64017-9714-4d4f-923f-737ce5be5fa7 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:32:14.630516 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-60b64017-9714-4d4f-923f-737ce5be5fa7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '30c1f31b50ce482290cd14d309e0b78f', 'request_id': 'req-60b64017-9714-4d4f-923f-737ce5be5fa7'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:32:14.640855 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-60b64017-9714-4d4f-923f-737ce5be5fa7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '30c1f31b50ce482290cd14d309e0b78f', 'request_id': 'req-60b64017-9714-4d4f-923f-737ce5be5fa7'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:32:14.646792 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-60b64017-9714-4d4f-923f-737ce5be5fa7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '30c1f31b50ce482290cd14d309e0b78f', 'request_id': 'req-60b64017-9714-4d4f-923f-737ce5be5fa7'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:32:14.794764 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-60b64017-9714-4d4f-923f-737ce5be5fa7 admin admin] The token cache is being invalidate because project 30c1f31b50ce482290cd14d309e0b78f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:32:14.794987 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-60b64017-9714-4d4f-923f-737ce5be5fa7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:32:14.795378 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-60b64017-9714-4d4f-923f-737ce5be5fa7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:32:14.797441 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 751/1510] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:32:14 2022] DELETE /identity/v3/projects/30c1f31b50ce482290cd14d309e0b78f => generated 0 bytes in 357 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:32:24.105246 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=62 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:32:24.792698 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=84,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:33:42.914557 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-012065ff-4eb7-4580-8fd1-944aa11083ce None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:33:42.915143 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-012065ff-4eb7-4580-8fd1-944aa11083ce None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:33:42.915143 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-012065ff-4eb7-4580-8fd1-944aa11083ce None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:33:42.916429 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 760/1511] 127.0.0.1 () {54 vars in 901 bytes} [Sun Sep 25 04:33:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:33:42.922103 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4c50a7e-7a82-49cc-8982-37433cf0cebe None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:33:42.922544 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4c50a7e-7a82-49cc-8982-37433cf0cebe None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:33:42.922681 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4c50a7e-7a82-49cc-8982-37433cf0cebe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:33:43.002942 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-a4c50a7e-7a82-49cc-8982-37433cf0cebe None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:33:43.004145 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-a4c50a7e-7a82-49cc-8982-37433cf0cebe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:33:43.109027 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 752/1512] 127.0.0.1 () {58 vars in 998 bytes} [Sun Sep 25 04:33:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:33:43.265522 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-012065ff-4eb7-4580-8fd1-944aa11083ce None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:33:43.267757 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-012065ff-4eb7-4580-8fd1-944aa11083ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:33:43.386614 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-012065ff-4eb7-4580-8fd1-944aa11083ce None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:33:43.392849 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10d9a210-d7a4-45ea-8b93-e24f5d142ea1 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e703b42c53974daaa608b0ccc3f018b1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'ResellerAdmin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:33:43.393448 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10d9a210-d7a4-45ea-8b93-e24f5d142ea1 service nova] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:33:43.393724 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10d9a210-d7a4-45ea-8b93-e24f5d142ea1 service nova] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:33:43.393996 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10d9a210-d7a4-45ea-8b93-e24f5d142ea1 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:33:43.401948 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10d9a210-d7a4-45ea-8b93-e24f5d142ea1 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:33:43.403511 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10d9a210-d7a4-45ea-8b93-e24f5d142ea1 service nova] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:33:43.413149 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 761/1513] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:33:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 149 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:33:43.894878 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0784f1b-228e-4648-aad4-e1901e0e58ee None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:33:43.895163 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0784f1b-228e-4648-aad4-e1901e0e58ee None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:33:43.895438 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0784f1b-228e-4648-aad4-e1901e0e58ee None None] PATH_INFO: `/v3` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:33:43.896860 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 753/1514] 10.222.0.55 () {54 vars in 925 bytes} [Sun Sep 25 04:33:43 2022] GET /identity/v3 => generated 255 bytes in 3 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:33:43.903998 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7629001d-96a9-43d6-a9f7-bceb3bbc6ed5 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:33:43.904452 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7629001d-96a9-43d6-a9f7-bceb3bbc6ed5 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:33:43.904803 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7629001d-96a9-43d6-a9f7-bceb3bbc6ed5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:33:43.971941 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-7629001d-96a9-43d6-a9f7-bceb3bbc6ed5 None None] MFA Rules not processed for user `676d04ef5a984edcaad63d9156568660`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:33:43.973288 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-7629001d-96a9-43d6-a9f7-bceb3bbc6ed5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:33:44.047352 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:33:44.053540 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 762/1515] 10.222.0.55 () {58 vars in 1015 bytes} [Sun Sep 25 04:33:43 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:33:44.505688 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea1d54f-a380-4307-ac19-29d67268c10e None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:33:44.506295 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea1d54f-a380-4307-ac19-29d67268c10e None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:33:44.506295 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea1d54f-a380-4307-ac19-29d67268c10e None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:33:44.507609 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 754/1516] 127.0.0.1 () {54 vars in 901 bytes} [Sun Sep 25 04:33:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:33:44.515337 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cb8a780-4896-421a-a888-6a11fa1da854 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:33:44.515619 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cb8a780-4896-421a-a888-6a11fa1da854 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:33:44.515890 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cb8a780-4896-421a-a888-6a11fa1da854 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:33:44.536059 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-3cb8a780-4896-421a-a888-6a11fa1da854 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:33:44.537271 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-3cb8a780-4896-421a-a888-6a11fa1da854 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:33:44.544991 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 763/1517] 127.0.0.1 () {58 vars in 998 bytes} [Sun Sep 25 04:33:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:33:44.695458 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dea1d54f-a380-4307-ac19-29d67268c10e None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:33:44.710526 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dea1d54f-a380-4307-ac19-29d67268c10e None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:33:44.716185 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bf7a6de-456f-49e3-a0a7-f66a548fed56 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e703b42c53974daaa608b0ccc3f018b1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'ResellerAdmin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:33:44.716841 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bf7a6de-456f-49e3-a0a7-f66a548fed56 service nova] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:33:44.717121 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bf7a6de-456f-49e3-a0a7-f66a548fed56 service nova] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:33:44.717392 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bf7a6de-456f-49e3-a0a7-f66a548fed56 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:33:44.726154 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bf7a6de-456f-49e3-a0a7-f66a548fed56 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:33:44.727758 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bf7a6de-456f-49e3-a0a7-f66a548fed56 service nova] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:33:44.737178 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 755/1518] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:33:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:33:53.235946 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-657bc396-c45e-4dba-902d-1042c7443f86 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:33:53.236459 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-657bc396-c45e-4dba-902d-1042c7443f86 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:33:53.236574 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-657bc396-c45e-4dba-902d-1042c7443f86 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:33:53.238130 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 764/1519] 10.222.0.25 () {54 vars in 961 bytes} [Sun Sep 25 04:33:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:33:53.242816 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f13b4c6-5a2e-4256-ba81-145b51ec883d None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:33:53.243250 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f13b4c6-5a2e-4256-ba81-145b51ec883d None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:33:53.243351 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f13b4c6-5a2e-4256-ba81-145b51ec883d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:33:53.263289 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-0f13b4c6-5a2e-4256-ba81-145b51ec883d None None] MFA Rules not processed for user `676d04ef5a984edcaad63d9156568660`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:33:53.264529 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-0f13b4c6-5a2e-4256-ba81-145b51ec883d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:33:53.273574 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 756/1520] 10.222.0.25 () {58 vars in 1058 bytes} [Sun Sep 25 04:33:53 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:33:53.283442 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-657bc396-c45e-4dba-902d-1042c7443f86 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:33:53.299252 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-657bc396-c45e-4dba-902d-1042c7443f86 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:33:53.304298 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbbd77ee-9a6b-45b6-9a77-271255985c22 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '676d04ef5a984edcaad63d9156568660', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:33:53.304943 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbbd77ee-9a6b-45b6-9a77-271255985c22 service placement] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:33:53.305220 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbbd77ee-9a6b-45b6-9a77-271255985c22 service placement] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:33:53.305488 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbbd77ee-9a6b-45b6-9a77-271255985c22 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:33:53.313862 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbbd77ee-9a6b-45b6-9a77-271255985c22 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:33:53.315431 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbbd77ee-9a6b-45b6-9a77-271255985c22 service placement] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:33:53.325300 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 765/1521] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:33:53 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:33:56.239023 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a26d171-a999-49c5-831c-751a9428c5c1 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:33:56.239606 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a26d171-a999-49c5-831c-751a9428c5c1 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:33:56.239606 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a26d171-a999-49c5-831c-751a9428c5c1 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:33:56.240941 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 757/1522] 127.0.0.1 () {54 vars in 901 bytes} [Sun Sep 25 04:33:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:33:56.247082 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ed3fc30-e34c-4413-9bbc-d835b898ce90 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:33:56.247515 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ed3fc30-e34c-4413-9bbc-d835b898ce90 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:33:56.247726 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ed3fc30-e34c-4413-9bbc-d835b898ce90 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:33:56.268358 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-2ed3fc30-e34c-4413-9bbc-d835b898ce90 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:33:56.269596 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-2ed3fc30-e34c-4413-9bbc-d835b898ce90 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:33:56.277414 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 766/1523] 127.0.0.1 () {58 vars in 998 bytes} [Sun Sep 25 04:33:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:33:56.431035 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a26d171-a999-49c5-831c-751a9428c5c1 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:33:56.446282 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a26d171-a999-49c5-831c-751a9428c5c1 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:33:56.451806 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12c9306b-dc44-4f7c-837a-434c09ffd97c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e703b42c53974daaa608b0ccc3f018b1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'ResellerAdmin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:33:56.452668 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12c9306b-dc44-4f7c-837a-434c09ffd97c service nova] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:33:56.452668 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12c9306b-dc44-4f7c-837a-434c09ffd97c service nova] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:33:56.452941 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12c9306b-dc44-4f7c-837a-434c09ffd97c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:33:56.461430 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12c9306b-dc44-4f7c-837a-434c09ffd97c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:33:56.463009 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12c9306b-dc44-4f7c-837a-434c09ffd97c service nova] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:33:56.472948 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 758/1524] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:33:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:33:57.553367 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-206b48e3-0ea3-40b8-ab32-1724dc3561e7 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:33:57.553933 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-206b48e3-0ea3-40b8-ab32-1724dc3561e7 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:33:57.553933 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-206b48e3-0ea3-40b8-ab32-1724dc3561e7 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:33:57.555298 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 767/1525] 127.0.0.1 () {54 vars in 901 bytes} [Sun Sep 25 04:33:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:33:57.561619 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed11b46-1b47-4387-8eee-6ba2e3260aea None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:33:57.562200 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed11b46-1b47-4387-8eee-6ba2e3260aea None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:33:57.562200 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed11b46-1b47-4387-8eee-6ba2e3260aea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:33:57.581642 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-fed11b46-1b47-4387-8eee-6ba2e3260aea None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:33:57.582839 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-fed11b46-1b47-4387-8eee-6ba2e3260aea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:33:57.590857 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 759/1526] 127.0.0.1 () {58 vars in 998 bytes} [Sun Sep 25 04:33:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:33:57.743294 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-206b48e3-0ea3-40b8-ab32-1724dc3561e7 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:33:57.757731 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-206b48e3-0ea3-40b8-ab32-1724dc3561e7 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:33:57.763211 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd71a1df-65cd-4120-b533-eaa46d30882e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e703b42c53974daaa608b0ccc3f018b1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'ResellerAdmin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:33:57.763800 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd71a1df-65cd-4120-b533-eaa46d30882e service nova] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:33:57.764078 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd71a1df-65cd-4120-b533-eaa46d30882e service nova] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:33:57.764364 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd71a1df-65cd-4120-b533-eaa46d30882e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:33:57.772280 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd71a1df-65cd-4120-b533-eaa46d30882e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:33:57.773881 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd71a1df-65cd-4120-b533-eaa46d30882e service nova] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:33:57.783647 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 768/1527] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:33:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:34:01.990909 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc08c19-4b87-4756-816f-45657964791d None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:34:01.991459 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc08c19-4b87-4756-816f-45657964791d None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:34:01.991459 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc08c19-4b87-4756-816f-45657964791d None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:34:01.992862 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 760/1528] 127.0.0.1 () {54 vars in 917 bytes} [Sun Sep 25 04:34:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:34:01.998153 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32ec975e-0a10-4f9c-ae41-28947c3b2b77 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:34:01.999257 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32ec975e-0a10-4f9c-ae41-28947c3b2b77 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:34:01.999481 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32ec975e-0a10-4f9c-ae41-28947c3b2b77 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:34:02.019364 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-32ec975e-0a10-4f9c-ae41-28947c3b2b77 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:34:02.020551 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-32ec975e-0a10-4f9c-ae41-28947c3b2b77 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:34:02.028286 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 769/1529] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Sep 25 04:34:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:34:02.330037 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cc08c19-4b87-4756-816f-45657964791d None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:34:02.332006 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-3cc08c19-4b87-4756-816f-45657964791d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:34:02.365229 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:34:02.411021 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cc08c19-4b87-4756-816f-45657964791d None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:34:02.414611 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cee55a06-b904-46da-a958-95f9f4b2a986 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:34:02.415220 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cee55a06-b904-46da-a958-95f9f4b2a986 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:34:02.415503 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cee55a06-b904-46da-a958-95f9f4b2a986 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:34:02.415779 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cee55a06-b904-46da-a958-95f9f4b2a986 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:34:02.425171 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cee55a06-b904-46da-a958-95f9f4b2a986 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:34:02.426634 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cee55a06-b904-46da-a958-95f9f4b2a986 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:34:02.435985 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 761/1530] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:34:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:34:04.600540 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1db6d837-5617-4fe5-9160-dd04ac96dedf None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:34:04.600971 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1db6d837-5617-4fe5-9160-dd04ac96dedf None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:34:04.601117 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1db6d837-5617-4fe5-9160-dd04ac96dedf None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:34:04.602449 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 770/1531] 127.0.0.1 () {54 vars in 917 bytes} [Sun Sep 25 04:34:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:34:04.608336 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-270f27b4-9c02-48b9-a7b7-e30ef6c93247 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:34:04.608774 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-270f27b4-9c02-48b9-a7b7-e30ef6c93247 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:34:04.609080 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-270f27b4-9c02-48b9-a7b7-e30ef6c93247 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:34:04.628909 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-270f27b4-9c02-48b9-a7b7-e30ef6c93247 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:34:04.630098 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-270f27b4-9c02-48b9-a7b7-e30ef6c93247 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:34:04.637784 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 762/1532] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Sep 25 04:34:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 04:34:04.926700 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1db6d837-5617-4fe5-9160-dd04ac96dedf None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:34:04.938070 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1db6d837-5617-4fe5-9160-dd04ac96dedf None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:34:04.941591 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48d88927-7755-4b2a-bc09-bb97a107a19c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:34:04.942183 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48d88927-7755-4b2a-bc09-bb97a107a19c service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:34:04.942462 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48d88927-7755-4b2a-bc09-bb97a107a19c service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:34:04.942730 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48d88927-7755-4b2a-bc09-bb97a107a19c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:34:04.951655 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48d88927-7755-4b2a-bc09-bb97a107a19c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:34:04.953112 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48d88927-7755-4b2a-bc09-bb97a107a19c service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:34:04.962634 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 771/1533] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:34:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:34:08.690514 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-567e1bc5-454a-4976-b05f-782facf3f961 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:34:08.691053 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-567e1bc5-454a-4976-b05f-782facf3f961 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:34:08.691053 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-567e1bc5-454a-4976-b05f-782facf3f961 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:34:08.692416 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 763/1534] 127.0.0.1 () {54 vars in 917 bytes} [Sun Sep 25 04:34:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:34:08.697773 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-241faca3-638b-482d-acee-39d64ecbaf93 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:34:08.698179 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-241faca3-638b-482d-acee-39d64ecbaf93 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:34:08.698357 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-241faca3-638b-482d-acee-39d64ecbaf93 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:34:08.717948 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-241faca3-638b-482d-acee-39d64ecbaf93 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:34:08.719178 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-241faca3-638b-482d-acee-39d64ecbaf93 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:34:08.727065 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 772/1535] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Sep 25 04:34:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:34:09.032476 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-567e1bc5-454a-4976-b05f-782facf3f961 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:34:09.034170 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-567e1bc5-454a-4976-b05f-782facf3f961 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:34:09.112779 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-567e1bc5-454a-4976-b05f-782facf3f961 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:34:09.116325 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a98f2dd8-95b1-4361-940e-27552166e2dd service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ffbc136c77b84e8c87128f7247f4ec03', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:34:09.116959 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a98f2dd8-95b1-4361-940e-27552166e2dd service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:34:09.117238 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a98f2dd8-95b1-4361-940e-27552166e2dd service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:34:09.117510 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a98f2dd8-95b1-4361-940e-27552166e2dd service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:34:09.126357 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a98f2dd8-95b1-4361-940e-27552166e2dd service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:34:09.127774 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a98f2dd8-95b1-4361-940e-27552166e2dd service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:34:09.137018 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 764/1536] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:34:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:34:11.270219 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d095cf-0d39-4d19-a375-4b006fd8a77e None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:34:11.270815 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d095cf-0d39-4d19-a375-4b006fd8a77e None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:34:11.270815 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d095cf-0d39-4d19-a375-4b006fd8a77e None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:34:11.272091 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 773/1537] 127.0.0.1 () {54 vars in 917 bytes} [Sun Sep 25 04:34:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:34:11.277434 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5dc543a-2c8e-455a-94cc-e8fad42b6946 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:34:11.277858 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5dc543a-2c8e-455a-94cc-e8fad42b6946 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:34:11.277973 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5dc543a-2c8e-455a-94cc-e8fad42b6946 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:34:11.298288 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-c5dc543a-2c8e-455a-94cc-e8fad42b6946 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:34:11.299485 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-c5dc543a-2c8e-455a-94cc-e8fad42b6946 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:34:11.307360 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 765/1538] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Sep 25 04:34:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:34:11.611121 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2d095cf-0d39-4d19-a375-4b006fd8a77e None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:34:11.612877 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-b2d095cf-0d39-4d19-a375-4b006fd8a77e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:34:11.626524 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2d095cf-0d39-4d19-a375-4b006fd8a77e None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:34:11.630015 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72cfcd16-cafb-4c56-85e8-25022d35dcd5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ffbc136c77b84e8c87128f7247f4ec03', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:34:11.630600 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72cfcd16-cafb-4c56-85e8-25022d35dcd5 service cinder] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:34:11.630902 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72cfcd16-cafb-4c56-85e8-25022d35dcd5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:34:11.631170 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72cfcd16-cafb-4c56-85e8-25022d35dcd5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:34:11.640131 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72cfcd16-cafb-4c56-85e8-25022d35dcd5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:34:11.641559 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72cfcd16-cafb-4c56-85e8-25022d35dcd5 service cinder] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:34:11.650661 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 774/1539] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:34:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:34:21.292832 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=63 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:34:21.646601 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=37 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:35:36.369323 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4a0b4e4-9961-4c5e-adab-891587ccb3b7 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:36.370061 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4a0b4e4-9961-4c5e-adab-891587ccb3b7 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:36.370061 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4a0b4e4-9961-4c5e-adab-891587ccb3b7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:36.394259 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-e4a0b4e4-9961-4c5e-adab-891587ccb3b7 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:35:36.395450 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-e4a0b4e4-9961-4c5e-adab-891587ccb3b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:35:36.409093 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 766/1540] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:35:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 04:35:36.418665 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ac59744-f2cf-47b8-b630-f835bd24012f None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:36.419289 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ac59744-f2cf-47b8-b630-f835bd24012f None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:36.419289 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ac59744-f2cf-47b8-b630-f835bd24012f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:36.445502 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-0ac59744-f2cf-47b8-b630-f835bd24012f None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:35:36.446704 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-0ac59744-f2cf-47b8-b630-f835bd24012f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:35:36.459965 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 775/1541] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:35:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:35:36.472511 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4a0b4e4-9961-4c5e-adab-891587ccb3b7 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:36.499836 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4a0b4e4-9961-4c5e-adab-891587ccb3b7 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:36.513985 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c55fb78d-b51e-4e80-b117-107067ae92fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:36.515235 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c55fb78d-b51e-4e80-b117-107067ae92fc admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:36.515779 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c55fb78d-b51e-4e80-b117-107067ae92fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:36.516468 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c55fb78d-b51e-4e80-b117-107067ae92fc admin admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:36.517461 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c55fb78d-b51e-4e80-b117-107067ae92fc admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:35:36.518028 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c55fb78d-b51e-4e80-b117-107067ae92fc admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:36.520033 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c55fb78d-b51e-4e80-b117-107067ae92fc admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:36.535347 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 767/1542] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Sep 25 04:35:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 71 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:35:36.543973 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ac59744-f2cf-47b8-b630-f835bd24012f None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:36.558637 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ac59744-f2cf-47b8-b630-f835bd24012f None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:36.567912 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd1933f0-233f-40ee-a918-f1817254bc46 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:36.568548 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd1933f0-233f-40ee-a918-f1817254bc46 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:36.568847 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd1933f0-233f-40ee-a918-f1817254bc46 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:36.569120 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd1933f0-233f-40ee-a918-f1817254bc46 admin admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:36.569771 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd1933f0-233f-40ee-a918-f1817254bc46 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:36.571661 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd1933f0-233f-40ee-a918-f1817254bc46 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:36.638340 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 776/1543] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:35:36 2022] POST /identity/v3/projects => generated 385 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 04:35:36.643809 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c55fb78d-b51e-4e80-b117-107067ae92fc admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:36.657252 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-295f8c39-a952-4ea1-a828-299d67477e83 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:36.657885 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-295f8c39-a952-4ea1-a828-299d67477e83 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:36.658381 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-295f8c39-a952-4ea1-a828-299d67477e83 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:36.659695 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c55fb78d-b51e-4e80-b117-107067ae92fc admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:36.696499 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-295f8c39-a952-4ea1-a828-299d67477e83 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:35:36.698375 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-295f8c39-a952-4ea1-a828-299d67477e83 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:35:36.789045 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84e74d0b-dbc3-464f-b9a9-03d6bae44f33 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:36.789045 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e74d0b-dbc3-464f-b9a9-03d6bae44f33 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:36.789045 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e74d0b-dbc3-464f-b9a9-03d6bae44f33 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:36.789045 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e74d0b-dbc3-464f-b9a9-03d6bae44f33 admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:36.789530 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84e74d0b-dbc3-464f-b9a9-03d6bae44f33 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:36.789530 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84e74d0b-dbc3-464f-b9a9-03d6bae44f33 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:36.839785 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 768/1544] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:35:36 2022] POST /identity/v3/users => generated 293 bytes in 197 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:35:36.841280 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2df44b38-d903-4458-8494-9c2d31230a03 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:36.841565 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2df44b38-d903-4458-8494-9c2d31230a03 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:36.841840 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2df44b38-d903-4458-8494-9c2d31230a03 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:36.852187 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 777/1545] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:35:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 197 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:35:36.854087 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-648826dd-5ccc-49fa-9a4d-b16d389b83d1 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:36.854610 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-648826dd-5ccc-49fa-9a4d-b16d389b83d1 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:36.855105 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-648826dd-5ccc-49fa-9a4d-b16d389b83d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:36.868502 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-2df44b38-d903-4458-8494-9c2d31230a03 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:35:36.870104 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-2df44b38-d903-4458-8494-9c2d31230a03 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:35:36.892924 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 769/1546] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:35:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 53 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:35:36.894910 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b79c069-214c-414c-b328-a9afff81ba1d None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:36.895412 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b79c069-214c-414c-b328-a9afff81ba1d None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:36.897497 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b79c069-214c-414c-b328-a9afff81ba1d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:36.908039 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-648826dd-5ccc-49fa-9a4d-b16d389b83d1 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:35:36.910137 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-648826dd-5ccc-49fa-9a4d-b16d389b83d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:35:36.936467 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 778/1547] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:35:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:35:36.937117 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-5b79c069-214c-414c-b328-a9afff81ba1d None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:35:36.937257 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-648826dd-5ccc-49fa-9a4d-b16d389b83d1 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:36.938371 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-5b79c069-214c-414c-b328-a9afff81ba1d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:35:36.948424 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 770/1548] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:35:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 54 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:35:36.949773 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-648826dd-5ccc-49fa-9a4d-b16d389b83d1 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:36.950635 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42f34ffb-60ee-4969-ac55-4b731b465988 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:36.951223 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42f34ffb-60ee-4969-ac55-4b731b465988 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:36.952432 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42f34ffb-60ee-4969-ac55-4b731b465988 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:36.955808 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9e6d095-8c3e-4483-b25e-13755bc5f638 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:36.956500 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9e6d095-8c3e-4483-b25e-13755bc5f638 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:36.957064 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9e6d095-8c3e-4483-b25e-13755bc5f638 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:36.957342 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9e6d095-8c3e-4483-b25e-13755bc5f638 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:36.958012 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9e6d095-8c3e-4483-b25e-13755bc5f638 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:36.959624 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9e6d095-8c3e-4483-b25e-13755bc5f638 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:36.971883 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 779/1549] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:35:36 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:35:36.975183 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08adebae-6ea1-4c85-b13d-4b3cc1702a21 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:36.975183 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08adebae-6ea1-4c85-b13d-4b3cc1702a21 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:36.975183 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08adebae-6ea1-4c85-b13d-4b3cc1702a21 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:36.981509 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-42f34ffb-60ee-4969-ac55-4b731b465988 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:35:36.984683 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-42f34ffb-60ee-4969-ac55-4b731b465988 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:35:36.992867 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 771/1550] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:35:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:35:36.994051 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00be0de0-f1d4-4f9f-ac13-37895a8eec39 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:36.994191 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-08adebae-6ea1-4c85-b13d-4b3cc1702a21 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:35:36.994379 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00be0de0-f1d4-4f9f-ac13-37895a8eec39 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:36.994594 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00be0de0-f1d4-4f9f-ac13-37895a8eec39 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:36.995291 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-08adebae-6ea1-4c85-b13d-4b3cc1702a21 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:35:37.004462 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 780/1551] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:35:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:35:37.005752 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5aedc0d-5573-4065-b42a-a4fcd1377c6e None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:37.006037 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5aedc0d-5573-4065-b42a-a4fcd1377c6e None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:37.006308 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5aedc0d-5573-4065-b42a-a4fcd1377c6e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:37.015519 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-00be0de0-f1d4-4f9f-ac13-37895a8eec39 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:35:37.016743 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-00be0de0-f1d4-4f9f-ac13-37895a8eec39 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:35:37.024555 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 772/1552] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:35:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:35:37.025774 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23d0c4f1-e79c-4d16-8119-67ee793ba66c None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:37.026054 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23d0c4f1-e79c-4d16-8119-67ee793ba66c None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:37.026324 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23d0c4f1-e79c-4d16-8119-67ee793ba66c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:37.032487 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-a5aedc0d-5573-4065-b42a-a4fcd1377c6e None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:35:37.033906 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-a5aedc0d-5573-4065-b42a-a4fcd1377c6e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:35:37.041918 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 781/1553] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:35:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:35:37.043081 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a983d69-da90-407f-9516-e4bb4cc1abab None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:37.043360 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a983d69-da90-407f-9516-e4bb4cc1abab None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:37.043627 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a983d69-da90-407f-9516-e4bb4cc1abab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:37.052386 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-23d0c4f1-e79c-4d16-8119-67ee793ba66c None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:35:37.053692 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-23d0c4f1-e79c-4d16-8119-67ee793ba66c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:35:37.062347 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 773/1554] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:35:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:35:37.066030 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23d0c4f1-e79c-4d16-8119-67ee793ba66c None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:37.066135 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-8a983d69-da90-407f-9516-e4bb4cc1abab None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:35:37.066135 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-8a983d69-da90-407f-9516-e4bb4cc1abab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:35:37.072914 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23d0c4f1-e79c-4d16-8119-67ee793ba66c None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:37.073267 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 782/1555] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:35:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:35:37.074718 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b254ccc3-c0d6-4ad5-88c8-81672be8b698 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:37.075012 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b254ccc3-c0d6-4ad5-88c8-81672be8b698 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:37.075287 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b254ccc3-c0d6-4ad5-88c8-81672be8b698 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:37.078412 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7469bd4a-f084-4b0c-9115-039f603aaf9e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:37.079008 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7469bd4a-f084-4b0c-9115-039f603aaf9e admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:37.079284 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7469bd4a-f084-4b0c-9115-039f603aaf9e admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:37.079582 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7469bd4a-f084-4b0c-9115-039f603aaf9e admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:37.080219 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7469bd4a-f084-4b0c-9115-039f603aaf9e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:37.082100 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7469bd4a-f084-4b0c-9115-039f603aaf9e admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:37.101106 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 774/1556] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:35:37 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:35:37.101106 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7469bd4a-f084-4b0c-9115-039f603aaf9e admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:37.102094 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-b254ccc3-c0d6-4ad5-88c8-81672be8b698 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:35:37.103973 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-b254ccc3-c0d6-4ad5-88c8-81672be8b698 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:35:37.112410 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7469bd4a-f084-4b0c-9115-039f603aaf9e admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:37.113870 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 783/1557] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:35:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:35:37.114676 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b254ccc3-c0d6-4ad5-88c8-81672be8b698 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:37.120226 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a7f36c7-2d65-43eb-a06e-93d2f1f8d1c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:37.121592 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a7f36c7-2d65-43eb-a06e-93d2f1f8d1c6 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:37.121873 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a7f36c7-2d65-43eb-a06e-93d2f1f8d1c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:37.122307 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a7f36c7-2d65-43eb-a06e-93d2f1f8d1c6 admin admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:37.122948 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a7f36c7-2d65-43eb-a06e-93d2f1f8d1c6 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:35:37.123254 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a7f36c7-2d65-43eb-a06e-93d2f1f8d1c6 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:37.124536 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b254ccc3-c0d6-4ad5-88c8-81672be8b698 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:37.125315 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a7f36c7-2d65-43eb-a06e-93d2f1f8d1c6 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:37.129287 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9d8fe28-16bc-47db-942d-dc284e31b2af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:37.129949 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9d8fe28-16bc-47db-942d-dc284e31b2af admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:37.130231 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9d8fe28-16bc-47db-942d-dc284e31b2af admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:37.130501 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9d8fe28-16bc-47db-942d-dc284e31b2af admin admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:37.131140 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9d8fe28-16bc-47db-942d-dc284e31b2af admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:35:37.131445 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9d8fe28-16bc-47db-942d-dc284e31b2af admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:37.132446 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: WARNING py.warnings [None req-b9d8fe28-16bc-47db-942d-dc284e31b2af admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Sep 25 04:35:37.132446 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: warnings.warn(msg) Sep 25 04:35:37.132446 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: Sep 25 04:35:37.133353 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9d8fe28-16bc-47db-942d-dc284e31b2af admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:37.137737 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 775/1558] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Sep 25 04:35:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:35:37.138926 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd9acbb-da54-47f3-8241-e28176cb16d0 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:37.139209 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd9acbb-da54-47f3-8241-e28176cb16d0 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:37.140230 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd9acbb-da54-47f3-8241-e28176cb16d0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:37.146471 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 784/1559] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Sep 25 04:35:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:35:37.147667 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df8b2b2d-b136-4de0-a431-4f90852b059c None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:37.147959 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df8b2b2d-b136-4de0-a431-4f90852b059c None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:37.148453 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df8b2b2d-b136-4de0-a431-4f90852b059c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:37.161293 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-2dd9acbb-da54-47f3-8241-e28176cb16d0 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:35:37.162497 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-2dd9acbb-da54-47f3-8241-e28176cb16d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:35:37.169965 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-df8b2b2d-b136-4de0-a431-4f90852b059c None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:35:37.174018 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 776/1560] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:35:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:35:37.174171 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-df8b2b2d-b136-4de0-a431-4f90852b059c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:35:37.174286 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dd9acbb-da54-47f3-8241-e28176cb16d0 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:37.180042 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 785/1561] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:35:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:35:37.180900 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df8b2b2d-b136-4de0-a431-4f90852b059c None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:37.181029 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dd9acbb-da54-47f3-8241-e28176cb16d0 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:37.185702 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46c968c3-6365-4834-a6d7-e717cf7fab8b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:37.186786 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c968c3-6365-4834-a6d7-e717cf7fab8b admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:37.187219 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c968c3-6365-4834-a6d7-e717cf7fab8b admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:37.187671 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c968c3-6365-4834-a6d7-e717cf7fab8b admin admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:37.188311 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46c968c3-6365-4834-a6d7-e717cf7fab8b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:35:37.188612 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46c968c3-6365-4834-a6d7-e717cf7fab8b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:37.190217 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46c968c3-6365-4834-a6d7-e717cf7fab8b admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:37.191112 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df8b2b2d-b136-4de0-a431-4f90852b059c None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:37.195901 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fdd2857-e411-4413-a960-e6c8b341190f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:37.196494 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fdd2857-e411-4413-a960-e6c8b341190f admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:37.196806 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fdd2857-e411-4413-a960-e6c8b341190f admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:37.197087 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fdd2857-e411-4413-a960-e6c8b341190f admin admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:37.197725 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fdd2857-e411-4413-a960-e6c8b341190f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:35:37.198031 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fdd2857-e411-4413-a960-e6c8b341190f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:37.199629 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fdd2857-e411-4413-a960-e6c8b341190f admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:37.209577 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 777/1562] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Sep 25 04:35:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:35:37.209577 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46c968c3-6365-4834-a6d7-e717cf7fab8b admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:37.215131 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46c968c3-6365-4834-a6d7-e717cf7fab8b admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:37.226754 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 786/1563] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Sep 25 04:35:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:35:37.226944 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-093125c3-8f43-400b-93d8-750a9e4cad94 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:37.226944 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-093125c3-8f43-400b-93d8-750a9e4cad94 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:37.226944 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-093125c3-8f43-400b-93d8-750a9e4cad94 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:37.227198 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46140b6e-eea0-4c9e-a2a6-17be4b357170 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:37.227277 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-093125c3-8f43-400b-93d8-750a9e4cad94 admin admin] PATH_INFO: `/v3/projects/5ead2221f7f54d6cbea9fc086a089342/users/c50ba6f9a164463eae9a248936e7bf80/roles/8f6cc574056c4278bc9778f14dc4b411` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:37.227354 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46140b6e-eea0-4c9e-a2a6-17be4b357170 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:37.227354 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46140b6e-eea0-4c9e-a2a6-17be4b357170 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:37.236317 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-093125c3-8f43-400b-93d8-750a9e4cad94 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5ead2221f7f54d6cbea9fc086a089342, user_id=c50ba6f9a164463eae9a248936e7bf80, role_id=8f6cc574056c4278bc9778f14dc4b411)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:37.237390 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: WARNING py.warnings [None req-093125c3-8f43-400b-93d8-750a9e4cad94 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 25 04:35:37.237390 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: warnings.warn(msg) Sep 25 04:35:37.237390 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: Sep 25 04:35:37.239312 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-093125c3-8f43-400b-93d8-750a9e4cad94 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:37.243423 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-46140b6e-eea0-4c9e-a2a6-17be4b357170 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:35:37.244879 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-46140b6e-eea0-4c9e-a2a6-17be4b357170 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:35:37.254103 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 778/1564] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:35:37 2022] PUT /identity/v3/projects/5ead2221f7f54d6cbea9fc086a089342/users/c50ba6f9a164463eae9a248936e7bf80/roles/8f6cc574056c4278bc9778f14dc4b411 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:35:37.254263 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 787/1565] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:35:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:35:37.254929 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46140b6e-eea0-4c9e-a2a6-17be4b357170 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:37.255465 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-093125c3-8f43-400b-93d8-750a9e4cad94 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:37.267271 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46140b6e-eea0-4c9e-a2a6-17be4b357170 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:37.289585 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:35:37.306727 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:35:37.326538 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-093125c3-8f43-400b-93d8-750a9e4cad94 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:37.328119 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb576ba7-3247-4ca8-b558-d8ea4f2671bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:37.328768 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb576ba7-3247-4ca8-b558-d8ea4f2671bf admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:37.329049 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb576ba7-3247-4ca8-b558-d8ea4f2671bf admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:37.329314 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb576ba7-3247-4ca8-b558-d8ea4f2671bf admin admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:37.329938 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb576ba7-3247-4ca8-b558-d8ea4f2671bf admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:37.331414 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b76c73c-f792-443d-b471-72ef1784d9d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:37.332079 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb576ba7-3247-4ca8-b558-d8ea4f2671bf admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:37.332393 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b76c73c-f792-443d-b471-72ef1784d9d6 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:37.332708 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b76c73c-f792-443d-b471-72ef1784d9d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:37.332988 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b76c73c-f792-443d-b471-72ef1784d9d6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:37.333611 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b76c73c-f792-443d-b471-72ef1784d9d6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:37.334644 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: WARNING py.warnings [None req-6b76c73c-f792-443d-b471-72ef1784d9d6 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 25 04:35:37.334644 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: warnings.warn(msg) Sep 25 04:35:37.334644 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: Sep 25 04:35:37.335819 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b76c73c-f792-443d-b471-72ef1784d9d6 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:37.360655 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 788/1566] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:35:37 2022] POST /identity/v3/projects => generated 393 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:35:37.361712 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb576ba7-3247-4ca8-b558-d8ea4f2671bf admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:37.365996 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 779/1567] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:35:37 2022] POST /identity/v3/projects => generated 405 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:35:37.365996 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b76c73c-f792-443d-b471-72ef1784d9d6 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:37.427450 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb576ba7-3247-4ca8-b558-d8ea4f2671bf admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:37.431825 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b76c73c-f792-443d-b471-72ef1784d9d6 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:37.432270 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93cef81e-adee-455d-ad0d-391d26e327ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:37.432850 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93cef81e-adee-455d-ad0d-391d26e327ce admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:37.433146 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93cef81e-adee-455d-ad0d-391d26e327ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:37.433427 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93cef81e-adee-455d-ad0d-391d26e327ce admin admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:37.434068 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93cef81e-adee-455d-ad0d-391d26e327ce admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:35:37.434375 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93cef81e-adee-455d-ad0d-391d26e327ce admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:37.436005 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93cef81e-adee-455d-ad0d-391d26e327ce admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:37.436876 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-293e8bbb-2318-46de-93a8-ccb2b46ab960 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:37.437483 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-293e8bbb-2318-46de-93a8-ccb2b46ab960 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:37.437767 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-293e8bbb-2318-46de-93a8-ccb2b46ab960 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:37.438041 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-293e8bbb-2318-46de-93a8-ccb2b46ab960 admin admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:37.438681 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-293e8bbb-2318-46de-93a8-ccb2b46ab960 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:35:37.438988 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-293e8bbb-2318-46de-93a8-ccb2b46ab960 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:37.440571 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-293e8bbb-2318-46de-93a8-ccb2b46ab960 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:37.449157 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 789/1568] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Sep 25 04:35:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:35:37.450219 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93cef81e-adee-455d-ad0d-391d26e327ce admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:37.452372 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 780/1569] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Sep 25 04:35:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 89 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:35:37.453206 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-293e8bbb-2318-46de-93a8-ccb2b46ab960 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:37.460664 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93cef81e-adee-455d-ad0d-391d26e327ce admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:37.463320 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-293e8bbb-2318-46de-93a8-ccb2b46ab960 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:37.469338 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-684ba8e1-f615-459a-b485-c276712213c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:37.469338 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-684ba8e1-f615-459a-b485-c276712213c3 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:37.469338 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-684ba8e1-f615-459a-b485-c276712213c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:37.469338 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-684ba8e1-f615-459a-b485-c276712213c3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:37.469758 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-684ba8e1-f615-459a-b485-c276712213c3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:37.469869 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-684ba8e1-f615-459a-b485-c276712213c3 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:37.471682 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17efe80c-ce19-4462-96fd-be5b675b72af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:37.472287 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17efe80c-ce19-4462-96fd-be5b675b72af admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:37.472564 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17efe80c-ce19-4462-96fd-be5b675b72af admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:37.472854 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17efe80c-ce19-4462-96fd-be5b675b72af admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:37.473476 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17efe80c-ce19-4462-96fd-be5b675b72af admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:37.475357 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17efe80c-ce19-4462-96fd-be5b675b72af admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:37.499767 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 790/1570] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:35:37 2022] POST /identity/v3/projects => generated 379 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:35:37.501329 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-895ecd19-a539-45be-a40e-83d7d47e7e88 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:37.501617 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-895ecd19-a539-45be-a40e-83d7d47e7e88 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:37.501892 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-895ecd19-a539-45be-a40e-83d7d47e7e88 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:37.506494 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 781/1571] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:35:37 2022] POST /identity/v3/projects => generated 375 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:35:37.507198 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17efe80c-ce19-4462-96fd-be5b675b72af admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:37.521904 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-895ecd19-a539-45be-a40e-83d7d47e7e88 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:35:37.523112 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-895ecd19-a539-45be-a40e-83d7d47e7e88 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:35:37.569809 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17efe80c-ce19-4462-96fd-be5b675b72af admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:37.575001 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e459c9e0-298b-44f0-a3dd-4465085591e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:37.575773 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e459c9e0-298b-44f0-a3dd-4465085591e0 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:37.576165 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e459c9e0-298b-44f0-a3dd-4465085591e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:37.576538 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e459c9e0-298b-44f0-a3dd-4465085591e0 admin admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:37.577843 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e459c9e0-298b-44f0-a3dd-4465085591e0 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:35:37.578184 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e459c9e0-298b-44f0-a3dd-4465085591e0 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:37.580407 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e459c9e0-298b-44f0-a3dd-4465085591e0 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:37.583553 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 791/1572] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:35:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:35:37.584275 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-895ecd19-a539-45be-a40e-83d7d47e7e88 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:37.594310 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-895ecd19-a539-45be-a40e-83d7d47e7e88 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:37.603716 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 782/1573] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Sep 25 04:35:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 90 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:35:37.603716 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e459c9e0-298b-44f0-a3dd-4465085591e0 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:37.606179 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc68c062-f23b-45aa-a04d-91d2984a118f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:37.606828 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc68c062-f23b-45aa-a04d-91d2984a118f admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:37.607113 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc68c062-f23b-45aa-a04d-91d2984a118f admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:37.607391 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc68c062-f23b-45aa-a04d-91d2984a118f admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:37.608406 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc68c062-f23b-45aa-a04d-91d2984a118f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:37.608681 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e459c9e0-298b-44f0-a3dd-4465085591e0 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:37.609469 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: WARNING py.warnings [None req-cc68c062-f23b-45aa-a04d-91d2984a118f admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 25 04:35:37.609469 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: warnings.warn(msg) Sep 25 04:35:37.609469 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: Sep 25 04:35:37.610688 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc68c062-f23b-45aa-a04d-91d2984a118f admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:37.613968 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-771d31c9-72a9-46d5-b73a-a999a37f8e48 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:37.614564 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-771d31c9-72a9-46d5-b73a-a999a37f8e48 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:37.614849 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-771d31c9-72a9-46d5-b73a-a999a37f8e48 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:37.615131 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-771d31c9-72a9-46d5-b73a-a999a37f8e48 admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:37.615765 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-771d31c9-72a9-46d5-b73a-a999a37f8e48 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:37.617768 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-771d31c9-72a9-46d5-b73a-a999a37f8e48 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:37.632654 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 792/1574] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:35:37 2022] POST /identity/v3/users => generated 297 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:35:37.633450 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc68c062-f23b-45aa-a04d-91d2984a118f admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:37.640206 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 783/1575] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:35:37 2022] POST /identity/v3/users => generated 303 bytes in 43 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:35:37.641448 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-771d31c9-72a9-46d5-b73a-a999a37f8e48 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:37.646688 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc68c062-f23b-45aa-a04d-91d2984a118f admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:37.647815 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:35:37.652032 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-413dc9de-f99e-42c9-8d3d-c3e419fba1ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:37.652776 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-413dc9de-f99e-42c9-8d3d-c3e419fba1ff admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:37.653093 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-413dc9de-f99e-42c9-8d3d-c3e419fba1ff admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:37.653367 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-413dc9de-f99e-42c9-8d3d-c3e419fba1ff admin admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:37.654000 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-413dc9de-f99e-42c9-8d3d-c3e419fba1ff admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:37.654316 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-771d31c9-72a9-46d5-b73a-a999a37f8e48 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:37.656018 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-413dc9de-f99e-42c9-8d3d-c3e419fba1ff admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:37.660090 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb48c186-ab76-4107-81be-7b68056c1417 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:37.660724 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb48c186-ab76-4107-81be-7b68056c1417 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:37.661008 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb48c186-ab76-4107-81be-7b68056c1417 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:37.661285 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb48c186-ab76-4107-81be-7b68056c1417 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:37.661912 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb48c186-ab76-4107-81be-7b68056c1417 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:37.663790 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb48c186-ab76-4107-81be-7b68056c1417 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:37.679201 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:35:37.692108 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 793/1576] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:35:37 2022] POST /identity/v3/projects => generated 371 bytes in 59 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:35:37.692928 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-413dc9de-f99e-42c9-8d3d-c3e419fba1ff admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:37.694685 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 784/1577] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:35:37 2022] POST /identity/v3/projects => generated 371 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:35:37.695481 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb48c186-ab76-4107-81be-7b68056c1417 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:37.756587 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-413dc9de-f99e-42c9-8d3d-c3e419fba1ff admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:37.760532 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb48c186-ab76-4107-81be-7b68056c1417 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:37.761266 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc192e12-0d4a-49f5-9ee7-a793864cc9b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:37.761872 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc192e12-0d4a-49f5-9ee7-a793864cc9b3 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:37.762150 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc192e12-0d4a-49f5-9ee7-a793864cc9b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:37.762419 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc192e12-0d4a-49f5-9ee7-a793864cc9b3 admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:37.763038 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc192e12-0d4a-49f5-9ee7-a793864cc9b3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:37.765824 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc192e12-0d4a-49f5-9ee7-a793864cc9b3 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:37.765987 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-702f8b86-6a25-4607-9953-ec2e1efe3ea7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:37.766553 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-702f8b86-6a25-4607-9953-ec2e1efe3ea7 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:37.766837 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-702f8b86-6a25-4607-9953-ec2e1efe3ea7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:37.767113 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-702f8b86-6a25-4607-9953-ec2e1efe3ea7 admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:37.767733 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-702f8b86-6a25-4607-9953-ec2e1efe3ea7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:37.769675 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-702f8b86-6a25-4607-9953-ec2e1efe3ea7 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:37.787757 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 794/1578] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:35:37 2022] POST /identity/v3/users => generated 290 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:35:37.788552 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc192e12-0d4a-49f5-9ee7-a793864cc9b3 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:37.792235 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 785/1579] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:35:37 2022] POST /identity/v3/users => generated 288 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:35:37.795250 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-702f8b86-6a25-4607-9953-ec2e1efe3ea7 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:37.795250 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-702f8b86-6a25-4607-9953-ec2e1efe3ea7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:35:37.801039 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc192e12-0d4a-49f5-9ee7-a793864cc9b3 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:37.805546 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7b63ccb-8a0d-4462-8e52-4318b449fee9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:37.806142 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7b63ccb-8a0d-4462-8e52-4318b449fee9 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:37.806440 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7b63ccb-8a0d-4462-8e52-4318b449fee9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:37.806709 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7b63ccb-8a0d-4462-8e52-4318b449fee9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:37.807328 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7b63ccb-8a0d-4462-8e52-4318b449fee9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:37.809240 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7b63ccb-8a0d-4462-8e52-4318b449fee9 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:37.841016 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 795/1580] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:35:37 2022] POST /identity/v3/projects => generated 389 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:35:37.841016 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7b63ccb-8a0d-4462-8e52-4318b449fee9 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:37.856836 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-702f8b86-6a25-4607-9953-ec2e1efe3ea7 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:37.902067 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7b63ccb-8a0d-4462-8e52-4318b449fee9 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:37.906933 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7fa93dc-69c6-4f0d-a55c-03e6bb164d0c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:37.907557 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7fa93dc-69c6-4f0d-a55c-03e6bb164d0c service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:37.907724 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-344e3656-bcf2-4a9b-954f-5eb236b4f34b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:37.907840 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7fa93dc-69c6-4f0d-a55c-03e6bb164d0c service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:37.908095 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7fa93dc-69c6-4f0d-a55c-03e6bb164d0c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:37.908351 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-344e3656-bcf2-4a9b-954f-5eb236b4f34b admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:37.908670 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-344e3656-bcf2-4a9b-954f-5eb236b4f34b admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:37.908949 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-344e3656-bcf2-4a9b-954f-5eb236b4f34b admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:37.909641 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-344e3656-bcf2-4a9b-954f-5eb236b4f34b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:37.911314 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-344e3656-bcf2-4a9b-954f-5eb236b4f34b admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:37.917609 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7fa93dc-69c6-4f0d-a55c-03e6bb164d0c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:37.919045 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7fa93dc-69c6-4f0d-a55c-03e6bb164d0c service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:37.921349 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 796/1581] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:35:37 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:35:37.922149 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-344e3656-bcf2-4a9b-954f-5eb236b4f34b admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:37.931322 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-344e3656-bcf2-4a9b-954f-5eb236b4f34b admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:37.932220 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 786/1582] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:35:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 139 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:35:37.932999 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7fa93dc-69c6-4f0d-a55c-03e6bb164d0c service neutron] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:37.936573 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30cbebd2-97a0-436b-a2c4-5529556ebc8e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:37.937518 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30cbebd2-97a0-436b-a2c4-5529556ebc8e admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:37.937964 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30cbebd2-97a0-436b-a2c4-5529556ebc8e admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:37.938516 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30cbebd2-97a0-436b-a2c4-5529556ebc8e admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:37.939324 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30cbebd2-97a0-436b-a2c4-5529556ebc8e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:37.940946 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30cbebd2-97a0-436b-a2c4-5529556ebc8e admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:37.944758 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7fa93dc-69c6-4f0d-a55c-03e6bb164d0c service neutron] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:37.950037 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-346939ee-fbc8-4943-890a-620962112cdc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:37.950779 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-346939ee-fbc8-4943-890a-620962112cdc admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:37.951306 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-346939ee-fbc8-4943-890a-620962112cdc admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:37.951429 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 797/1583] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:35:37 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:35:37.951530 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-346939ee-fbc8-4943-890a-620962112cdc admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:37.952480 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30cbebd2-97a0-436b-a2c4-5529556ebc8e admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:37.952745 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-346939ee-fbc8-4943-890a-620962112cdc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:37.954978 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-346939ee-fbc8-4943-890a-620962112cdc admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:37.964577 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30cbebd2-97a0-436b-a2c4-5529556ebc8e admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:37.970616 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47346f33-cfbf-44b2-96e2-7d199641504c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:37.971547 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47346f33-cfbf-44b2-96e2-7d199641504c admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:37.971845 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47346f33-cfbf-44b2-96e2-7d199641504c admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:37.972124 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47346f33-cfbf-44b2-96e2-7d199641504c admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:37.973109 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47346f33-cfbf-44b2-96e2-7d199641504c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:37.976272 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47346f33-cfbf-44b2-96e2-7d199641504c admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:37.978468 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 787/1584] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:35:37 2022] POST /identity/v3/users => generated 286 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:35:37.979176 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-346939ee-fbc8-4943-890a-620962112cdc admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:37.992332 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-346939ee-fbc8-4943-890a-620962112cdc admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:37.997546 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d432c6df-9aa4-442b-821c-2da67f7ffecf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:37.998168 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d432c6df-9aa4-442b-821c-2da67f7ffecf admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:37.998455 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d432c6df-9aa4-442b-821c-2da67f7ffecf admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:37.999258 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d432c6df-9aa4-442b-821c-2da67f7ffecf admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:38.000092 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d432c6df-9aa4-442b-821c-2da67f7ffecf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:38.001778 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d432c6df-9aa4-442b-821c-2da67f7ffecf admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:38.005290 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 798/1585] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:35:37 2022] POST /identity/v3/users => generated 286 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:35:38.005290 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47346f33-cfbf-44b2-96e2-7d199641504c admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:38.007912 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:35:38.013798 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 788/1586] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:35:37 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:35:38.015333 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d432c6df-9aa4-442b-821c-2da67f7ffecf admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:38.020349 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47346f33-cfbf-44b2-96e2-7d199641504c admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:38.027433 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c771c8ed-dcf1-47e4-96f8-527660c753ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:38.028192 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c771c8ed-dcf1-47e4-96f8-527660c753ac admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:38.028444 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d432c6df-9aa4-442b-821c-2da67f7ffecf admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:38.028569 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c771c8ed-dcf1-47e4-96f8-527660c753ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:38.028853 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c771c8ed-dcf1-47e4-96f8-527660c753ac admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:38.029536 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c771c8ed-dcf1-47e4-96f8-527660c753ac admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:38.031475 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c771c8ed-dcf1-47e4-96f8-527660c753ac admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:38.033576 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39296952-299c-4e0c-9b8d-791ce61ec9a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:38.034188 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39296952-299c-4e0c-9b8d-791ce61ec9a8 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:38.034477 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39296952-299c-4e0c-9b8d-791ce61ec9a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:38.034761 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39296952-299c-4e0c-9b8d-791ce61ec9a8 admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:38.035397 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39296952-299c-4e0c-9b8d-791ce61ec9a8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:38.037507 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39296952-299c-4e0c-9b8d-791ce61ec9a8 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:38.045148 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 799/1587] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:35:38 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:35:38.046317 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c771c8ed-dcf1-47e4-96f8-527660c753ac admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:38.060479 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c771c8ed-dcf1-47e4-96f8-527660c753ac admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:38.060882 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 789/1588] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:35:38 2022] POST /identity/v3/users => generated 295 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:35:38.062693 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39296952-299c-4e0c-9b8d-791ce61ec9a8 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:38.068602 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2417c053-8ea3-41ee-9066-9a345b6d5ba1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:38.070580 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2417c053-8ea3-41ee-9066-9a345b6d5ba1 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:38.071142 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2417c053-8ea3-41ee-9066-9a345b6d5ba1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:38.071763 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2417c053-8ea3-41ee-9066-9a345b6d5ba1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:38.072624 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2417c053-8ea3-41ee-9066-9a345b6d5ba1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:38.074163 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39296952-299c-4e0c-9b8d-791ce61ec9a8 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:38.074996 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2417c053-8ea3-41ee-9066-9a345b6d5ba1 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:38.079857 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d8bb105-d1c8-40b6-b2db-677e06ebebae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:38.080471 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d8bb105-d1c8-40b6-b2db-677e06ebebae admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:38.080788 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d8bb105-d1c8-40b6-b2db-677e06ebebae admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:38.081064 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d8bb105-d1c8-40b6-b2db-677e06ebebae admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:38.081965 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d8bb105-d1c8-40b6-b2db-677e06ebebae admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:38.083574 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d8bb105-d1c8-40b6-b2db-677e06ebebae admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:38.090050 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 800/1589] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:35:38 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:35:38.091458 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2417c053-8ea3-41ee-9066-9a345b6d5ba1 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:38.093924 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 790/1590] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:35:38 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:35:38.097735 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d8bb105-d1c8-40b6-b2db-677e06ebebae admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:38.103282 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2417c053-8ea3-41ee-9066-9a345b6d5ba1 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:38.108449 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcfd2784-50dc-4c6f-971a-9f1b05689b7f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:38.109120 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcfd2784-50dc-4c6f-971a-9f1b05689b7f admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:38.109502 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcfd2784-50dc-4c6f-971a-9f1b05689b7f admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:38.109678 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d8bb105-d1c8-40b6-b2db-677e06ebebae admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:38.109789 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcfd2784-50dc-4c6f-971a-9f1b05689b7f admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:38.110407 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcfd2784-50dc-4c6f-971a-9f1b05689b7f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:38.112007 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcfd2784-50dc-4c6f-971a-9f1b05689b7f admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:38.115849 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6d248a3-2c55-4d94-8be0-1ccd346b8a0b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:38.116383 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6d248a3-2c55-4d94-8be0-1ccd346b8a0b admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:38.116768 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6d248a3-2c55-4d94-8be0-1ccd346b8a0b admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:38.117069 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6d248a3-2c55-4d94-8be0-1ccd346b8a0b admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:38.117966 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:35:38.118158 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6d248a3-2c55-4d94-8be0-1ccd346b8a0b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:38.120253 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6d248a3-2c55-4d94-8be0-1ccd346b8a0b admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:38.127671 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 801/1591] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:35:38 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:35:38.127671 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcfd2784-50dc-4c6f-971a-9f1b05689b7f admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:38.132707 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 791/1592] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:35:38 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:35:38.133576 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6d248a3-2c55-4d94-8be0-1ccd346b8a0b admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:38.141191 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcfd2784-50dc-4c6f-971a-9f1b05689b7f admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:38.144629 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6d248a3-2c55-4d94-8be0-1ccd346b8a0b admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:38.146363 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e25c13e-30a5-4f2d-ac58-7beb8219e81b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:38.146974 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e25c13e-30a5-4f2d-ac58-7beb8219e81b admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:38.147250 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e25c13e-30a5-4f2d-ac58-7beb8219e81b admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:38.147521 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e25c13e-30a5-4f2d-ac58-7beb8219e81b admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:38.148186 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e25c13e-30a5-4f2d-ac58-7beb8219e81b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:38.149931 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e25c13e-30a5-4f2d-ac58-7beb8219e81b admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:38.150084 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbd5f7d1-8ea6-4660-8498-f80cc611dcc8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:38.150692 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbd5f7d1-8ea6-4660-8498-f80cc611dcc8 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:38.151005 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbd5f7d1-8ea6-4660-8498-f80cc611dcc8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:38.151458 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbd5f7d1-8ea6-4660-8498-f80cc611dcc8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:38.152474 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbd5f7d1-8ea6-4660-8498-f80cc611dcc8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:38.154095 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbd5f7d1-8ea6-4660-8498-f80cc611dcc8 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:38.162012 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 802/1593] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:35:38 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:35:38.165934 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e25c13e-30a5-4f2d-ac58-7beb8219e81b admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:38.168201 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 792/1594] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:35:38 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:35:38.169042 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbd5f7d1-8ea6-4660-8498-f80cc611dcc8 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:38.178507 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e25c13e-30a5-4f2d-ac58-7beb8219e81b admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:38.183299 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fc717da-8b87-4f6c-826b-3030851e373c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:38.183502 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbd5f7d1-8ea6-4660-8498-f80cc611dcc8 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:38.183843 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc717da-8b87-4f6c-826b-3030851e373c admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:38.184130 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc717da-8b87-4f6c-826b-3030851e373c admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:38.184400 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc717da-8b87-4f6c-826b-3030851e373c admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:38.185073 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fc717da-8b87-4f6c-826b-3030851e373c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:38.186725 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fc717da-8b87-4f6c-826b-3030851e373c admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:38.189220 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d9a6a0d-4750-4cef-a46d-3f5addbca509 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:38.189889 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d9a6a0d-4750-4cef-a46d-3f5addbca509 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:38.190309 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d9a6a0d-4750-4cef-a46d-3f5addbca509 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:38.190592 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d9a6a0d-4750-4cef-a46d-3f5addbca509 admin admin] PATH_INFO: `/v3/projects/f6f23cf0323f48069e21acf059cbd1f0/users/2666d4774776498bbdbce539296e76a9/roles/8f6cc574056c4278bc9778f14dc4b411` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:38.197003 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 803/1595] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:35:38 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:35:38.197800 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fc717da-8b87-4f6c-826b-3030851e373c admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:38.208525 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d9a6a0d-4750-4cef-a46d-3f5addbca509 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f6f23cf0323f48069e21acf059cbd1f0, user_id=2666d4774776498bbdbce539296e76a9, role_id=8f6cc574056c4278bc9778f14dc4b411)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:38.208782 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fc717da-8b87-4f6c-826b-3030851e373c admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:38.212467 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d9a6a0d-4750-4cef-a46d-3f5addbca509 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:38.214366 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bb04fb9-3d1c-443f-a9c3-a20c93ef1adc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:38.215034 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bb04fb9-3d1c-443f-a9c3-a20c93ef1adc admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:38.215316 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bb04fb9-3d1c-443f-a9c3-a20c93ef1adc admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:38.215591 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bb04fb9-3d1c-443f-a9c3-a20c93ef1adc admin admin] PATH_INFO: `/v3/projects/b01d6b07a05345deb81df6c18d28a181/users/ed0fdf2718d341c68c403a3f073ad024/roles/8f6cc574056c4278bc9778f14dc4b411` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:38.231301 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 793/1596] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:35:38 2022] PUT /identity/v3/projects/f6f23cf0323f48069e21acf059cbd1f0/users/2666d4774776498bbdbce539296e76a9/roles/8f6cc574056c4278bc9778f14dc4b411 => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:35:38.231301 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d9a6a0d-4750-4cef-a46d-3f5addbca509 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:38.238244 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bb04fb9-3d1c-443f-a9c3-a20c93ef1adc admin admin] RBAC: Authorizing `identity:create_grant(project_id=b01d6b07a05345deb81df6c18d28a181, user_id=ed0fdf2718d341c68c403a3f073ad024, role_id=8f6cc574056c4278bc9778f14dc4b411)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:38.241292 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bb04fb9-3d1c-443f-a9c3-a20c93ef1adc admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:38.268253 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 804/1597] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:35:38 2022] PUT /identity/v3/projects/b01d6b07a05345deb81df6c18d28a181/users/ed0fdf2718d341c68c403a3f073ad024/roles/8f6cc574056c4278bc9778f14dc4b411 => generated 0 bytes in 60 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:35:38.268253 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bb04fb9-3d1c-443f-a9c3-a20c93ef1adc admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:38.303791 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d9a6a0d-4750-4cef-a46d-3f5addbca509 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:38.360771 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bb04fb9-3d1c-443f-a9c3-a20c93ef1adc admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:38.367753 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e352aff5-45c3-4b48-a929-a59d2ef6118a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:38.368535 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e352aff5-45c3-4b48-a929-a59d2ef6118a admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:38.368872 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e352aff5-45c3-4b48-a929-a59d2ef6118a admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:38.369221 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e352aff5-45c3-4b48-a929-a59d2ef6118a admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:38.369912 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e352aff5-45c3-4b48-a929-a59d2ef6118a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:38.372083 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e352aff5-45c3-4b48-a929-a59d2ef6118a admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:38.392356 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 805/1598] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:35:38 2022] GET /identity/v3/roles => generated 1420 bytes in 130 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:35:38.392356 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e352aff5-45c3-4b48-a929-a59d2ef6118a admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:38.393058 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3315981b-635c-420f-8e89-8d403e79ae29 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:38.393982 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3315981b-635c-420f-8e89-8d403e79ae29 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:38.394445 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3315981b-635c-420f-8e89-8d403e79ae29 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:38.394881 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3315981b-635c-420f-8e89-8d403e79ae29 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:38.396012 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3315981b-635c-420f-8e89-8d403e79ae29 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:38.397969 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3315981b-635c-420f-8e89-8d403e79ae29 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:38.407097 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e352aff5-45c3-4b48-a929-a59d2ef6118a admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:38.410701 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 794/1599] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:35:38 2022] GET /identity/v3/roles => generated 1420 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:35:38.411947 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3315981b-635c-420f-8e89-8d403e79ae29 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:38.413367 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ef0b75d-c37c-4743-a89e-48595ddcbdf9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:38.417267 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ef0b75d-c37c-4743-a89e-48595ddcbdf9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:38.418135 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ef0b75d-c37c-4743-a89e-48595ddcbdf9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:38.418652 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ef0b75d-c37c-4743-a89e-48595ddcbdf9 admin admin] PATH_INFO: `/v3/projects/80e5b6ce14ab47dfac99cc80153d7735/users/1d1bf78eb0a94bab884b1df13c081096/roles/8f6cc574056c4278bc9778f14dc4b411` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:38.423700 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3315981b-635c-420f-8e89-8d403e79ae29 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:38.430265 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4411c1b8-2dff-4b4d-9f79-136da1240886 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:38.430935 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4411c1b8-2dff-4b4d-9f79-136da1240886 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:38.431216 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4411c1b8-2dff-4b4d-9f79-136da1240886 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:38.431637 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4411c1b8-2dff-4b4d-9f79-136da1240886 admin admin] PATH_INFO: `/v3/projects/e1108f4c9da246cb9b4b9f89833ceeda/users/16281c79f73e4063b52066966c393148/roles/8f6cc574056c4278bc9778f14dc4b411` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:38.451447 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ef0b75d-c37c-4743-a89e-48595ddcbdf9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=80e5b6ce14ab47dfac99cc80153d7735, user_id=1d1bf78eb0a94bab884b1df13c081096, role_id=8f6cc574056c4278bc9778f14dc4b411)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:38.452379 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4411c1b8-2dff-4b4d-9f79-136da1240886 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e1108f4c9da246cb9b4b9f89833ceeda, user_id=16281c79f73e4063b52066966c393148, role_id=8f6cc574056c4278bc9778f14dc4b411)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:38.454862 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ef0b75d-c37c-4743-a89e-48595ddcbdf9 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:38.455316 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4411c1b8-2dff-4b4d-9f79-136da1240886 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:38.472196 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 806/1600] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:35:38 2022] PUT /identity/v3/projects/80e5b6ce14ab47dfac99cc80153d7735/users/1d1bf78eb0a94bab884b1df13c081096/roles/8f6cc574056c4278bc9778f14dc4b411 => generated 0 bytes in 83 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:35:38.472920 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ef0b75d-c37c-4743-a89e-48595ddcbdf9 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:38.481948 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 795/1601] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:35:38 2022] PUT /identity/v3/projects/e1108f4c9da246cb9b4b9f89833ceeda/users/16281c79f73e4063b52066966c393148/roles/8f6cc574056c4278bc9778f14dc4b411 => generated 0 bytes in 70 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:35:38.484067 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-631cdc09-b0d4-4a8e-bfb1-0f30ba7307f1 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:38.484328 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-631cdc09-b0d4-4a8e-bfb1-0f30ba7307f1 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:38.484601 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-631cdc09-b0d4-4a8e-bfb1-0f30ba7307f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:38.495518 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:35:38.509344 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-631cdc09-b0d4-4a8e-bfb1-0f30ba7307f1 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:35:38.510588 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-631cdc09-b0d4-4a8e-bfb1-0f30ba7307f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:35:38.571064 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ef0b75d-c37c-4743-a89e-48595ddcbdf9 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:38.580620 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 796/1602] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:35:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:35:38.582360 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea93592-6c77-47b6-8048-a94c93120593 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:38.582654 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea93592-6c77-47b6-8048-a94c93120593 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:38.582926 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea93592-6c77-47b6-8048-a94c93120593 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:38.607793 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:35:38.614423 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-eea93592-6c77-47b6-8048-a94c93120593 None None] MFA Rules not processed for user `ed0fdf2718d341c68c403a3f073ad024`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:35:38.616782 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-eea93592-6c77-47b6-8048-a94c93120593 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:35:38.641953 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a740c9f-a918-499b-9d18-c95e1e46d29e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:38.643453 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a740c9f-a918-499b-9d18-c95e1e46d29e admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:38.645859 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a740c9f-a918-499b-9d18-c95e1e46d29e admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:38.646536 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a740c9f-a918-499b-9d18-c95e1e46d29e admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:38.647950 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a740c9f-a918-499b-9d18-c95e1e46d29e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:38.650008 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a740c9f-a918-499b-9d18-c95e1e46d29e admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:38.666271 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 807/1603] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:35:38 2022] GET /identity/v3/roles => generated 1420 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:35:38.667138 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a740c9f-a918-499b-9d18-c95e1e46d29e admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:38.679044 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a740c9f-a918-499b-9d18-c95e1e46d29e admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:38.684304 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba2e8bd8-fe85-42b4-840f-a2f7207ce1d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:38.685288 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba2e8bd8-fe85-42b4-840f-a2f7207ce1d2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:38.685752 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba2e8bd8-fe85-42b4-840f-a2f7207ce1d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:38.686870 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba2e8bd8-fe85-42b4-840f-a2f7207ce1d2 admin admin] PATH_INFO: `/v3/projects/8f27e933cfea467784e3621b1d1b7093/users/dabc3b39ba454df88179d56dce082443/roles/8f6cc574056c4278bc9778f14dc4b411` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:38.704112 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 797/1604] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:35:38 2022] POST /identity/v3/auth/tokens => generated 3282 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:35:38.705105 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eea93592-6c77-47b6-8048-a94c93120593 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:38.715044 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba2e8bd8-fe85-42b4-840f-a2f7207ce1d2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8f27e933cfea467784e3621b1d1b7093, user_id=dabc3b39ba454df88179d56dce082443, role_id=8f6cc574056c4278bc9778f14dc4b411)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:38.719706 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba2e8bd8-fe85-42b4-840f-a2f7207ce1d2 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:38.720173 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eea93592-6c77-47b6-8048-a94c93120593 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:38.725876 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0865b34e-dedf-409f-bc37-2f7e930bf51e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:38.726634 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0865b34e-dedf-409f-bc37-2f7e930bf51e admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:38.726920 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0865b34e-dedf-409f-bc37-2f7e930bf51e admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:38.727196 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0865b34e-dedf-409f-bc37-2f7e930bf51e admin admin] PATH_INFO: `/v3/projects/957087b9343a41ddbdb7c28fda2de2f3/users/71b36f0919a7427c87ce7cd76e85e4fd/roles/8f6cc574056c4278bc9778f14dc4b411` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:38.735603 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 808/1605] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:35:38 2022] PUT /identity/v3/projects/8f27e933cfea467784e3621b1d1b7093/users/dabc3b39ba454df88179d56dce082443/roles/8f6cc574056c4278bc9778f14dc4b411 => generated 0 bytes in 69 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:35:38.747396 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71d19198-3f85-44da-8512-f4fbd1cabc81 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:38.747991 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71d19198-3f85-44da-8512-f4fbd1cabc81 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:38.748350 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71d19198-3f85-44da-8512-f4fbd1cabc81 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:38.749503 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0865b34e-dedf-409f-bc37-2f7e930bf51e admin admin] RBAC: Authorizing `identity:create_grant(project_id=957087b9343a41ddbdb7c28fda2de2f3, user_id=71b36f0919a7427c87ce7cd76e85e4fd, role_id=8f6cc574056c4278bc9778f14dc4b411)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:38.757267 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0865b34e-dedf-409f-bc37-2f7e930bf51e admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:38.769888 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-71d19198-3f85-44da-8512-f4fbd1cabc81 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:35:38.770086 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-71d19198-3f85-44da-8512-f4fbd1cabc81 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:35:38.780548 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 798/1606] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:35:38 2022] PUT /identity/v3/projects/957087b9343a41ddbdb7c28fda2de2f3/users/71b36f0919a7427c87ce7cd76e85e4fd/roles/8f6cc574056c4278bc9778f14dc4b411 => generated 0 bytes in 75 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:35:38.782372 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3063599-d826-4e77-a4e4-91ff1f20fabb None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:38.782898 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3063599-d826-4e77-a4e4-91ff1f20fabb None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:38.783385 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3063599-d826-4e77-a4e4-91ff1f20fabb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:38.817693 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-c3063599-d826-4e77-a4e4-91ff1f20fabb None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:35:38.820037 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-c3063599-d826-4e77-a4e4-91ff1f20fabb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:35:38.857586 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 809/1607] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:35:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:35:38.862936 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71d19198-3f85-44da-8512-f4fbd1cabc81 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:38.922286 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 799/1608] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:35:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:35:38.924113 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3063599-d826-4e77-a4e4-91ff1f20fabb None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:38.939173 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3063599-d826-4e77-a4e4-91ff1f20fabb None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:38.942612 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71d19198-3f85-44da-8512-f4fbd1cabc81 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:38.946353 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bb26e6b-139c-432a-aa25-a3514b8f4f93 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:38.947077 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb26e6b-139c-432a-aa25-a3514b8f4f93 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:38.947461 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb26e6b-139c-432a-aa25-a3514b8f4f93 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:38.948822 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb26e6b-139c-432a-aa25-a3514b8f4f93 admin admin] PATH_INFO: `/v3/projects/a2ac515b97884375b3e85c6e54de81bc/users/e7354aa144df4ad69cd7d12f11b1d8a5/roles/8f6cc574056c4278bc9778f14dc4b411` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:38.953367 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c4d7ffd-fd81-4c92-9a31-726f702db0db service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:38.954513 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c4d7ffd-fd81-4c92-9a31-726f702db0db service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:38.954996 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c4d7ffd-fd81-4c92-9a31-726f702db0db service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:38.955503 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c4d7ffd-fd81-4c92-9a31-726f702db0db service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:38.958328 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:35:38.967186 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bb26e6b-139c-432a-aa25-a3514b8f4f93 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a2ac515b97884375b3e85c6e54de81bc, user_id=e7354aa144df4ad69cd7d12f11b1d8a5, role_id=8f6cc574056c4278bc9778f14dc4b411)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:38.970941 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bb26e6b-139c-432a-aa25-a3514b8f4f93 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:38.971481 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c4d7ffd-fd81-4c92-9a31-726f702db0db service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:38.973132 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c4d7ffd-fd81-4c92-9a31-726f702db0db service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:38.986174 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 810/1609] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:35:38 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 127 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:35:38.986825 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c4d7ffd-fd81-4c92-9a31-726f702db0db service neutron] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:38.992393 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 800/1610] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:35:38 2022] PUT /identity/v3/projects/a2ac515b97884375b3e85c6e54de81bc/users/e7354aa144df4ad69cd7d12f11b1d8a5/roles/8f6cc574056c4278bc9778f14dc4b411 => generated 0 bytes in 66 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:35:38.992393 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0283eacb-43e9-4ecd-85f8-24d746f92e1a None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:38.992393 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0283eacb-43e9-4ecd-85f8-24d746f92e1a None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:38.993441 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0283eacb-43e9-4ecd-85f8-24d746f92e1a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:39.004171 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c4d7ffd-fd81-4c92-9a31-726f702db0db service neutron] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:39.048842 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-0283eacb-43e9-4ecd-85f8-24d746f92e1a None None] MFA Rules not processed for user `dabc3b39ba454df88179d56dce082443`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:35:39.051042 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-0283eacb-43e9-4ecd-85f8-24d746f92e1a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:35:39.064533 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:35:39.084575 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88790fa5-7710-4d90-8e10-86d0e207e05b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:39.085517 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88790fa5-7710-4d90-8e10-86d0e207e05b admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:39.086068 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88790fa5-7710-4d90-8e10-86d0e207e05b admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:39.086453 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88790fa5-7710-4d90-8e10-86d0e207e05b admin admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:39.088204 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88790fa5-7710-4d90-8e10-86d0e207e05b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:39.091030 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88790fa5-7710-4d90-8e10-86d0e207e05b admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:39.138984 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 811/1611] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:35:38 2022] POST /identity/v3/projects => generated 405 bytes in 152 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:35:39.142345 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b8d5b4-28cc-4c84-8865-84d45fba1497 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:39.142784 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b8d5b4-28cc-4c84-8865-84d45fba1497 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:39.143224 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b8d5b4-28cc-4c84-8865-84d45fba1497 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:39.196738 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-37b8d5b4-28cc-4c84-8865-84d45fba1497 None None] MFA Rules not processed for user `71b36f0919a7427c87ce7cd76e85e4fd`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:35:39.198528 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-37b8d5b4-28cc-4c84-8865-84d45fba1497 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:35:39.211688 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 801/1612] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:35:38 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 222 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:35:39.212881 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0283eacb-43e9-4ecd-85f8-24d746f92e1a None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:39.291979 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 812/1613] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:35:39 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 152 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:35:39.293038 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37b8d5b4-28cc-4c84-8865-84d45fba1497 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:39.333605 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0283eacb-43e9-4ecd-85f8-24d746f92e1a None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:39.379100 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37b8d5b4-28cc-4c84-8865-84d45fba1497 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:39.386106 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d11b7d89-cb69-4bc8-a690-778bb9df04e0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:39.386998 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d11b7d89-cb69-4bc8-a690-778bb9df04e0 service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:39.387282 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d11b7d89-cb69-4bc8-a690-778bb9df04e0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:39.387556 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d11b7d89-cb69-4bc8-a690-778bb9df04e0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:39.411305 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fd19d81-0b06-4c12-97e7-0a0cb3bef4ee service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:39.412450 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fd19d81-0b06-4c12-97e7-0a0cb3bef4ee service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:39.414106 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fd19d81-0b06-4c12-97e7-0a0cb3bef4ee service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:39.414536 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fd19d81-0b06-4c12-97e7-0a0cb3bef4ee service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:39.528777 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:35:39.549623 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d11b7d89-cb69-4bc8-a690-778bb9df04e0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:39.551119 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d11b7d89-cb69-4bc8-a690-778bb9df04e0 service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:39.568626 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fd19d81-0b06-4c12-97e7-0a0cb3bef4ee service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:39.570131 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fd19d81-0b06-4c12-97e7-0a0cb3bef4ee service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:39.571725 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 813/1614] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:35:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 279 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:35:39.573316 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b997831-cb95-4be9-9670-2fe0fd582133 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:39.573739 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b997831-cb95-4be9-9670-2fe0fd582133 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:39.574085 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b997831-cb95-4be9-9670-2fe0fd582133 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:39.588184 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 802/1615] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:35:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 376 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:35:39.588936 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fd19d81-0b06-4c12-97e7-0a0cb3bef4ee service neutron] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:39.609317 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-5b997831-cb95-4be9-9670-2fe0fd582133 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:35:39.612124 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-5b997831-cb95-4be9-9670-2fe0fd582133 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:35:39.612620 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fd19d81-0b06-4c12-97e7-0a0cb3bef4ee service neutron] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:39.622704 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61054d5f-6caa-4b5a-b48e-7f249948e742 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:39.623336 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61054d5f-6caa-4b5a-b48e-7f249948e742 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:39.623657 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61054d5f-6caa-4b5a-b48e-7f249948e742 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:39.624089 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61054d5f-6caa-4b5a-b48e-7f249948e742 admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:39.624974 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61054d5f-6caa-4b5a-b48e-7f249948e742 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:39.626938 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61054d5f-6caa-4b5a-b48e-7f249948e742 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:39.633779 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 814/1616] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:35:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 61 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:35:39.634934 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b997831-cb95-4be9-9670-2fe0fd582133 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:39.649078 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b997831-cb95-4be9-9670-2fe0fd582133 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:39.659107 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10773e28-659e-4c40-9d33-dad2d3f4eb6a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:39.661208 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10773e28-659e-4c40-9d33-dad2d3f4eb6a admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:39.661769 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10773e28-659e-4c40-9d33-dad2d3f4eb6a admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:39.662429 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10773e28-659e-4c40-9d33-dad2d3f4eb6a admin admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:39.666842 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10773e28-659e-4c40-9d33-dad2d3f4eb6a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:39.671052 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10773e28-659e-4c40-9d33-dad2d3f4eb6a admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:39.671967 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 803/1617] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:35:39 2022] POST /identity/v3/users => generated 309 bytes in 83 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:35:39.673020 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61054d5f-6caa-4b5a-b48e-7f249948e742 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:39.677102 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:35:39.687989 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61054d5f-6caa-4b5a-b48e-7f249948e742 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:39.695668 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58a9f99e-82f4-4af0-bdd9-1429b5c68df7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:39.696369 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58a9f99e-82f4-4af0-bdd9-1429b5c68df7 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:39.696807 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58a9f99e-82f4-4af0-bdd9-1429b5c68df7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:39.697237 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58a9f99e-82f4-4af0-bdd9-1429b5c68df7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:39.697964 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58a9f99e-82f4-4af0-bdd9-1429b5c68df7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:39.700142 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58a9f99e-82f4-4af0-bdd9-1429b5c68df7 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:39.730596 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 815/1618] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:35:39 2022] POST /identity/v3/projects => generated 371 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:35:39.732025 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10773e28-659e-4c40-9d33-dad2d3f4eb6a admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:39.746209 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 804/1619] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:35:39 2022] POST /identity/v3/projects => generated 371 bytes in 65 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:35:39.746209 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58a9f99e-82f4-4af0-bdd9-1429b5c68df7 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:39.876952 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10773e28-659e-4c40-9d33-dad2d3f4eb6a admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:39.881301 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58a9f99e-82f4-4af0-bdd9-1429b5c68df7 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:39.886361 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc7aee65-2c1d-40fa-87fb-56e607862a28 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:39.887224 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc7aee65-2c1d-40fa-87fb-56e607862a28 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:39.887722 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc7aee65-2c1d-40fa-87fb-56e607862a28 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:39.888188 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc7aee65-2c1d-40fa-87fb-56e607862a28 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:40.024455 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e252b69-0973-458c-8f55-ca21d9a19ebf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:40.029528 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e252b69-0973-458c-8f55-ca21d9a19ebf admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:40.029528 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e252b69-0973-458c-8f55-ca21d9a19ebf admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:40.029528 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e252b69-0973-458c-8f55-ca21d9a19ebf admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:40.029528 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e252b69-0973-458c-8f55-ca21d9a19ebf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:40.034297 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e252b69-0973-458c-8f55-ca21d9a19ebf admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:40.040417 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc7aee65-2c1d-40fa-87fb-56e607862a28 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:40.045141 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc7aee65-2c1d-40fa-87fb-56e607862a28 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:40.066052 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 816/1620] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:35:39 2022] GET /identity/v3/roles => generated 1420 bytes in 335 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:35:40.067599 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e252b69-0973-458c-8f55-ca21d9a19ebf admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:40.083357 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 805/1621] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:35:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 344 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:35:40.083728 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc7aee65-2c1d-40fa-87fb-56e607862a28 service neutron] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:40.156503 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc7aee65-2c1d-40fa-87fb-56e607862a28 service neutron] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:40.166563 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e252b69-0973-458c-8f55-ca21d9a19ebf admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:40.167257 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad1986c1-b492-4c06-8f10-1a347e4085dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:40.167257 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad1986c1-b492-4c06-8f10-1a347e4085dc admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:40.167257 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad1986c1-b492-4c06-8f10-1a347e4085dc admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:40.167257 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad1986c1-b492-4c06-8f10-1a347e4085dc admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:40.168829 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad1986c1-b492-4c06-8f10-1a347e4085dc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:40.168829 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad1986c1-b492-4c06-8f10-1a347e4085dc admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:40.169051 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71c46b0e-968d-4440-b7e5-543521943620 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:40.169051 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71c46b0e-968d-4440-b7e5-543521943620 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:40.169051 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71c46b0e-968d-4440-b7e5-543521943620 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:40.169051 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71c46b0e-968d-4440-b7e5-543521943620 admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:40.169608 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71c46b0e-968d-4440-b7e5-543521943620 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:40.169608 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71c46b0e-968d-4440-b7e5-543521943620 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:40.201767 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 817/1623] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:35:40 2022] POST /identity/v3/users => generated 292 bytes in 135 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:35:40.205374 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 806/1622] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:35:40 2022] POST /identity/v3/users => generated 292 bytes in 114 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:35:40.206890 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad1986c1-b492-4c06-8f10-1a347e4085dc admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:40.207936 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71c46b0e-968d-4440-b7e5-543521943620 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:40.227161 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad1986c1-b492-4c06-8f10-1a347e4085dc admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:40.228088 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71c46b0e-968d-4440-b7e5-543521943620 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:40.235176 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3726b98-e8b1-4aed-99fa-b42c64feaf57 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:40.236131 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3726b98-e8b1-4aed-99fa-b42c64feaf57 admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:40.236513 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3726b98-e8b1-4aed-99fa-b42c64feaf57 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:40.237118 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3726b98-e8b1-4aed-99fa-b42c64feaf57 admin admin] PATH_INFO: `/v3/projects/c7beb085fe6b41d19fbdf454a0981ff9/users/f0541b72570d4835b4841f98c826ebe8/roles/c51fe7c5ad804040879acfb54269455f` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:40.241092 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d64555e8-926c-45d5-ace5-265f07356901 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:40.241979 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d64555e8-926c-45d5-ace5-265f07356901 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:40.242460 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d64555e8-926c-45d5-ace5-265f07356901 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:40.242907 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d64555e8-926c-45d5-ace5-265f07356901 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:40.243732 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d64555e8-926c-45d5-ace5-265f07356901 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:40.245606 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d64555e8-926c-45d5-ace5-265f07356901 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:40.272296 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 807/1624] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:35:40 2022] GET /identity/v3/roles => generated 1420 bytes in 66 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:35:40.273797 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d64555e8-926c-45d5-ace5-265f07356901 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:40.292987 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d64555e8-926c-45d5-ace5-265f07356901 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:40.300059 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3726b98-e8b1-4aed-99fa-b42c64feaf57 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c7beb085fe6b41d19fbdf454a0981ff9, user_id=f0541b72570d4835b4841f98c826ebe8, role_id=c51fe7c5ad804040879acfb54269455f)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:40.304849 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3726b98-e8b1-4aed-99fa-b42c64feaf57 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:40.308580 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e867b558-0646-47dc-93c5-7a56a96343af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:40.310068 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e867b558-0646-47dc-93c5-7a56a96343af admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:40.310363 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e867b558-0646-47dc-93c5-7a56a96343af admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:40.310645 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e867b558-0646-47dc-93c5-7a56a96343af admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:40.311488 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e867b558-0646-47dc-93c5-7a56a96343af admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:40.313220 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e867b558-0646-47dc-93c5-7a56a96343af admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:40.331033 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 808/1625] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:35:40 2022] GET /identity/v3/roles => generated 1420 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:35:40.331033 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e867b558-0646-47dc-93c5-7a56a96343af admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:40.341778 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e867b558-0646-47dc-93c5-7a56a96343af admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:40.343012 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 818/1626] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:35:40 2022] PUT /identity/v3/projects/c7beb085fe6b41d19fbdf454a0981ff9/users/f0541b72570d4835b4841f98c826ebe8/roles/c51fe7c5ad804040879acfb54269455f => generated 0 bytes in 136 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:35:40.344399 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3726b98-e8b1-4aed-99fa-b42c64feaf57 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:40.359005 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:35:40.414917 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:35:40.439132 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3726b98-e8b1-4aed-99fa-b42c64feaf57 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:40.447795 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f283379-46d1-4952-b77f-a5984a80e732 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:40.449040 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f283379-46d1-4952-b77f-a5984a80e732 admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:40.449946 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abd65374-612a-477a-8879-0dfedd38b800 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:40.451307 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f283379-46d1-4952-b77f-a5984a80e732 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:40.451576 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abd65374-612a-477a-8879-0dfedd38b800 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:40.451764 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f283379-46d1-4952-b77f-a5984a80e732 admin admin] PATH_INFO: `/v3/projects/aa02bff12b2f4d82870b443a017b1c18/users/c9c815de3e9e4e548e092db41154c082/roles/c51fe7c5ad804040879acfb54269455f` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:40.452488 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abd65374-612a-477a-8879-0dfedd38b800 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:40.453102 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abd65374-612a-477a-8879-0dfedd38b800 admin admin] PATH_INFO: `/v3/projects/14c065d1976144cc9c960e2968da9c14/users/8f57caa464e6420ebe1d6c766edc5f5c/roles/c51fe7c5ad804040879acfb54269455f` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:40.472986 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f283379-46d1-4952-b77f-a5984a80e732 admin admin] RBAC: Authorizing `identity:create_grant(project_id=aa02bff12b2f4d82870b443a017b1c18, user_id=c9c815de3e9e4e548e092db41154c082, role_id=c51fe7c5ad804040879acfb54269455f)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:40.477186 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f283379-46d1-4952-b77f-a5984a80e732 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:40.500787 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 819/1627] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:35:40 2022] PUT /identity/v3/projects/aa02bff12b2f4d82870b443a017b1c18/users/c9c815de3e9e4e548e092db41154c082/roles/c51fe7c5ad804040879acfb54269455f => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:35:40.501085 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abd65374-612a-477a-8879-0dfedd38b800 admin admin] RBAC: Authorizing `identity:create_grant(project_id=14c065d1976144cc9c960e2968da9c14, user_id=8f57caa464e6420ebe1d6c766edc5f5c, role_id=c51fe7c5ad804040879acfb54269455f)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:40.501846 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f283379-46d1-4952-b77f-a5984a80e732 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:40.504033 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abd65374-612a-477a-8879-0dfedd38b800 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:40.536935 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 809/1628] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:35:40 2022] PUT /identity/v3/projects/14c065d1976144cc9c960e2968da9c14/users/8f57caa464e6420ebe1d6c766edc5f5c/roles/c51fe7c5ad804040879acfb54269455f => generated 0 bytes in 208 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:35:40.538296 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abd65374-612a-477a-8879-0dfedd38b800 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:40.610694 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f283379-46d1-4952-b77f-a5984a80e732 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:40.661342 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abd65374-612a-477a-8879-0dfedd38b800 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:40.675127 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e782dae-4ea6-41b7-8819-e44c84d886e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:40.677351 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e782dae-4ea6-41b7-8819-e44c84d886e2 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:40.677701 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e782dae-4ea6-41b7-8819-e44c84d886e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:40.677987 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e782dae-4ea6-41b7-8819-e44c84d886e2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:40.679887 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e782dae-4ea6-41b7-8819-e44c84d886e2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:40.684137 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e782dae-4ea6-41b7-8819-e44c84d886e2 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:40.724119 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 810/1629] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:35:40 2022] GET /identity/v3/roles => generated 1420 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:35:40.726577 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e782dae-4ea6-41b7-8819-e44c84d886e2 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:40.747317 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e782dae-4ea6-41b7-8819-e44c84d886e2 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:40.765898 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de60e8ba-9f83-43a6-8381-cb6264730d58 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:40.767892 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de60e8ba-9f83-43a6-8381-cb6264730d58 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:40.768281 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de60e8ba-9f83-43a6-8381-cb6264730d58 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:40.768697 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de60e8ba-9f83-43a6-8381-cb6264730d58 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:40.769329 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4056975-c1da-4f4c-82fc-e7a76d5d14ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:40.769762 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de60e8ba-9f83-43a6-8381-cb6264730d58 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:40.772281 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de60e8ba-9f83-43a6-8381-cb6264730d58 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:40.775439 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4056975-c1da-4f4c-82fc-e7a76d5d14ff admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:40.775998 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4056975-c1da-4f4c-82fc-e7a76d5d14ff admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:40.777231 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4056975-c1da-4f4c-82fc-e7a76d5d14ff admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:40.780926 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4056975-c1da-4f4c-82fc-e7a76d5d14ff admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:40.786126 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 820/1630] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:35:40 2022] GET /identity/v3/roles => generated 1420 bytes in 285 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:35:40.787052 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de60e8ba-9f83-43a6-8381-cb6264730d58 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:40.793799 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4056975-c1da-4f4c-82fc-e7a76d5d14ff admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:40.813773 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de60e8ba-9f83-43a6-8381-cb6264730d58 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:40.833938 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d51f2ab-8101-469e-adcc-091444a511bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:40.836042 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d51f2ab-8101-469e-adcc-091444a511bf admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:40.836602 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d51f2ab-8101-469e-adcc-091444a511bf admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:40.838304 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d51f2ab-8101-469e-adcc-091444a511bf admin admin] PATH_INFO: `/v3/projects/aa02bff12b2f4d82870b443a017b1c18/users/c9c815de3e9e4e548e092db41154c082/roles/c51fe7c5ad804040879acfb54269455f` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:40.840450 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 811/1631] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:35:40 2022] GET /identity/v3/roles => generated 1420 bytes in 116 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:35:40.845638 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d51f2ab-8101-469e-adcc-091444a511bf admin admin] RBAC: Authorizing `identity:create_grant(project_id=aa02bff12b2f4d82870b443a017b1c18, user_id=c9c815de3e9e4e548e092db41154c082, role_id=c51fe7c5ad804040879acfb54269455f)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:40.848687 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d51f2ab-8101-469e-adcc-091444a511bf admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:40.859926 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4056975-c1da-4f4c-82fc-e7a76d5d14ff admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:40.876006 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 821/1632] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:35:40 2022] PUT /identity/v3/projects/aa02bff12b2f4d82870b443a017b1c18/users/c9c815de3e9e4e548e092db41154c082/roles/c51fe7c5ad804040879acfb54269455f => generated 0 bytes in 90 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:35:40.876897 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d51f2ab-8101-469e-adcc-091444a511bf admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:40.884050 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4056975-c1da-4f4c-82fc-e7a76d5d14ff admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:40.982576 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5a2479e-b4c5-4a19-97d5-752159c33412 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:40.984188 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5a2479e-b4c5-4a19-97d5-752159c33412 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:40.984616 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5a2479e-b4c5-4a19-97d5-752159c33412 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:40.985020 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5a2479e-b4c5-4a19-97d5-752159c33412 admin admin] PATH_INFO: `/v3/projects/c7beb085fe6b41d19fbdf454a0981ff9/users/f0541b72570d4835b4841f98c826ebe8/roles/c51fe7c5ad804040879acfb54269455f` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:40.987915 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5a2479e-b4c5-4a19-97d5-752159c33412 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c7beb085fe6b41d19fbdf454a0981ff9, user_id=f0541b72570d4835b4841f98c826ebe8, role_id=c51fe7c5ad804040879acfb54269455f)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:41.000606 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5a2479e-b4c5-4a19-97d5-752159c33412 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:41.042438 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 812/1633] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:35:40 2022] PUT /identity/v3/projects/c7beb085fe6b41d19fbdf454a0981ff9/users/f0541b72570d4835b4841f98c826ebe8/roles/c51fe7c5ad804040879acfb54269455f => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:35:41.044052 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7dc6b3b-85f6-446b-8ee7-9e18e396ad6e None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:41.044606 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7dc6b3b-85f6-446b-8ee7-9e18e396ad6e None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:41.048850 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7dc6b3b-85f6-446b-8ee7-9e18e396ad6e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:41.057447 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d51f2ab-8101-469e-adcc-091444a511bf admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:41.068665 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:35:41.130583 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-f7dc6b3b-85f6-446b-8ee7-9e18e396ad6e None None] MFA Rules not processed for user `c9c815de3e9e4e548e092db41154c082`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:35:41.132196 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-f7dc6b3b-85f6-446b-8ee7-9e18e396ad6e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:35:41.139880 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:35:41.240078 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edf6d82b-c323-49b3-ade9-35d90226f40b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:41.242231 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edf6d82b-c323-49b3-ade9-35d90226f40b admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:41.244520 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edf6d82b-c323-49b3-ade9-35d90226f40b admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:41.248172 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edf6d82b-c323-49b3-ade9-35d90226f40b admin admin] PATH_INFO: `/v3/projects/14c065d1976144cc9c960e2968da9c14/users/8f57caa464e6420ebe1d6c766edc5f5c/roles/c51fe7c5ad804040879acfb54269455f` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:41.251931 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edf6d82b-c323-49b3-ade9-35d90226f40b admin admin] RBAC: Authorizing `identity:create_grant(project_id=14c065d1976144cc9c960e2968da9c14, user_id=8f57caa464e6420ebe1d6c766edc5f5c, role_id=c51fe7c5ad804040879acfb54269455f)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:41.259449 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edf6d82b-c323-49b3-ade9-35d90226f40b admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:41.263292 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 813/1634] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:35:41 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 220 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:35:41.265152 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8c0af7d-fa75-45cc-a60e-dd6b33499f85 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:41.265559 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8c0af7d-fa75-45cc-a60e-dd6b33499f85 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:41.265929 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8c0af7d-fa75-45cc-a60e-dd6b33499f85 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:41.280906 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 822/1635] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:35:40 2022] PUT /identity/v3/projects/14c065d1976144cc9c960e2968da9c14/users/8f57caa464e6420ebe1d6c766edc5f5c/roles/c51fe7c5ad804040879acfb54269455f => generated 0 bytes in 404 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:35:41.282224 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edf6d82b-c323-49b3-ade9-35d90226f40b admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:41.325529 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-f8c0af7d-fa75-45cc-a60e-dd6b33499f85 None None] MFA Rules not processed for user `f0541b72570d4835b4841f98c826ebe8`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:35:41.327307 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-f8c0af7d-fa75-45cc-a60e-dd6b33499f85 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:35:41.355624 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edf6d82b-c323-49b3-ade9-35d90226f40b admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:41.362726 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3e4c614-176e-4334-87dd-cc2ea3a94be1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ffbc136c77b84e8c87128f7247f4ec03', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:41.363431 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3e4c614-176e-4334-87dd-cc2ea3a94be1 service cinder] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:41.363764 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3e4c614-176e-4334-87dd-cc2ea3a94be1 service cinder] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:41.364046 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3e4c614-176e-4334-87dd-cc2ea3a94be1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:41.466922 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 814/1636] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:35:41 2022] POST /identity/v3/auth/tokens => generated 3349 bytes in 200 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:35:41.466922 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35b53dda-2e8e-43b1-97da-6baf193117a4 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:41.466922 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35b53dda-2e8e-43b1-97da-6baf193117a4 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:41.466922 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35b53dda-2e8e-43b1-97da-6baf193117a4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:41.468001 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3e4c614-176e-4334-87dd-cc2ea3a94be1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:41.470339 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3e4c614-176e-4334-87dd-cc2ea3a94be1 service cinder] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:41.508772 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 823/1637] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:35:41 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 227 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:35:41.509954 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3e4c614-176e-4334-87dd-cc2ea3a94be1 service cinder] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:41.511636 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-35b53dda-2e8e-43b1-97da-6baf193117a4 None None] MFA Rules not processed for user `8f57caa464e6420ebe1d6c766edc5f5c`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:35:41.513243 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-35b53dda-2e8e-43b1-97da-6baf193117a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:35:41.534076 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3e4c614-176e-4334-87dd-cc2ea3a94be1 service cinder] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:41.542814 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa89c40d-a74f-4cc3-b74d-62554e9a6b70 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ffbc136c77b84e8c87128f7247f4ec03', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:41.543730 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa89c40d-a74f-4cc3-b74d-62554e9a6b70 service cinder] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:41.544176 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa89c40d-a74f-4cc3-b74d-62554e9a6b70 service cinder] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:41.544605 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa89c40d-a74f-4cc3-b74d-62554e9a6b70 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:41.566442 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa89c40d-a74f-4cc3-b74d-62554e9a6b70 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:41.568165 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa89c40d-a74f-4cc3-b74d-62554e9a6b70 service cinder] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:41.599810 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 824/1638] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:35:41 2022] GET /identity/v3/auth/tokens => generated 3349 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:35:41.610742 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 815/1639] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:35:41 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:35:41.697006 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa89c40d-a74f-4cc3-b74d-62554e9a6b70 service cinder] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:41.721354 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa89c40d-a74f-4cc3-b74d-62554e9a6b70 service cinder] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:41.736105 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b6a9b76-cb90-43e4-9f6d-deca5bab7722 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ffbc136c77b84e8c87128f7247f4ec03', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:41.737593 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b6a9b76-cb90-43e4-9f6d-deca5bab7722 service cinder] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:41.738145 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b6a9b76-cb90-43e4-9f6d-deca5bab7722 service cinder] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:41.738491 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b6a9b76-cb90-43e4-9f6d-deca5bab7722 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:41.765213 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b6a9b76-cb90-43e4-9f6d-deca5bab7722 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:41.767022 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b6a9b76-cb90-43e4-9f6d-deca5bab7722 service cinder] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:41.780898 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 825/1640] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:35:41 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:35:41.972575 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35b53dda-2e8e-43b1-97da-6baf193117a4 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:41.994904 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35b53dda-2e8e-43b1-97da-6baf193117a4 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:42.008868 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-032af5be-cd1b-4b50-a346-f9280e200bf4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ffbc136c77b84e8c87128f7247f4ec03', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:42.012044 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-032af5be-cd1b-4b50-a346-f9280e200bf4 service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:42.013253 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-032af5be-cd1b-4b50-a346-f9280e200bf4 service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:42.013654 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-032af5be-cd1b-4b50-a346-f9280e200bf4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:42.048057 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:35:42.161414 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-032af5be-cd1b-4b50-a346-f9280e200bf4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:42.163374 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-032af5be-cd1b-4b50-a346-f9280e200bf4 service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:42.175689 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 816/1641] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:35:41 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 204 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:35:42.184934 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b6a9b76-cb90-43e4-9f6d-deca5bab7722 service cinder] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:42.247569 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:35:42.358849 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b6a9b76-cb90-43e4-9f6d-deca5bab7722 service cinder] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:42.376162 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04ef4052-9d9a-4fe1-9653-fcbf3d954784 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:42.378219 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ef4052-9d9a-4fe1-9653-fcbf3d954784 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:42.378643 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ef4052-9d9a-4fe1-9653-fcbf3d954784 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:42.379824 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ef4052-9d9a-4fe1-9653-fcbf3d954784 admin admin] PATH_INFO: `/v3/users/ed0fdf2718d341c68c403a3f073ad024` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:42.383081 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04ef4052-9d9a-4fe1-9653-fcbf3d954784 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ed0fdf2718d341c68c403a3f073ad024)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:42.391436 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04ef4052-9d9a-4fe1-9653-fcbf3d954784 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:42.591248 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-04ef4052-9d9a-4fe1-9653-fcbf3d954784 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ed0fdf2718d341c68c403a3f073ad024'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:35:42.604931 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-04ef4052-9d9a-4fe1-9653-fcbf3d954784 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ed0fdf2718d341c68c403a3f073ad024'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:35:42.636926 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-04ef4052-9d9a-4fe1-9653-fcbf3d954784 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ed0fdf2718d341c68c403a3f073ad024'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:35:42.640116 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-04ef4052-9d9a-4fe1-9653-fcbf3d954784 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ed0fdf2718d341c68c403a3f073ad024'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:35:42.658097 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-04ef4052-9d9a-4fe1-9653-fcbf3d954784 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ed0fdf2718d341c68c403a3f073ad024'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:35:42.674045 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 826/1642] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:35:42 2022] DELETE /identity/v3/users/ed0fdf2718d341c68c403a3f073ad024 => generated 0 bytes in 489 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:35:42.680216 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-383fdf6b-b4f2-4cf7-b708-344fa4f69d3b None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:42.680737 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-383fdf6b-b4f2-4cf7-b708-344fa4f69d3b None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:42.680841 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-383fdf6b-b4f2-4cf7-b708-344fa4f69d3b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:42.765952 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-383fdf6b-b4f2-4cf7-b708-344fa4f69d3b None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:35:42.771694 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-383fdf6b-b4f2-4cf7-b708-344fa4f69d3b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:35:42.772195 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04ef4052-9d9a-4fe1-9653-fcbf3d954784 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:42.773982 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-04ef4052-9d9a-4fe1-9653-fcbf3d954784 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:35:42.925618 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 817/1643] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:35:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 248 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:35:42.933597 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04ef4052-9d9a-4fe1-9653-fcbf3d954784 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:42.939647 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cd517bb-e199-42a6-9279-6ff0cdd2c9ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:42.940396 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cd517bb-e199-42a6-9279-6ff0cdd2c9ec admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:42.940719 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cd517bb-e199-42a6-9279-6ff0cdd2c9ec admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:42.940998 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cd517bb-e199-42a6-9279-6ff0cdd2c9ec admin admin] PATH_INFO: `/v3/users/dabc3b39ba454df88179d56dce082443` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:42.942396 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cd517bb-e199-42a6-9279-6ff0cdd2c9ec admin admin] RBAC: Authorizing `identity:delete_user(user_id=dabc3b39ba454df88179d56dce082443)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:42.944428 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cd517bb-e199-42a6-9279-6ff0cdd2c9ec admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:43.023732 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=82,keystone:DELETE=17,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:35:43.026547 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-4cd517bb-e199-42a6-9279-6ff0cdd2c9ec admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'dabc3b39ba454df88179d56dce082443'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:35:43.033348 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-383fdf6b-b4f2-4cf7-b708-344fa4f69d3b None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:43.035411 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-383fdf6b-b4f2-4cf7-b708-344fa4f69d3b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:35:43.058239 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-4cd517bb-e199-42a6-9279-6ff0cdd2c9ec admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'dabc3b39ba454df88179d56dce082443'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:35:43.104341 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-4cd517bb-e199-42a6-9279-6ff0cdd2c9ec admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'dabc3b39ba454df88179d56dce082443'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:35:43.106277 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-4cd517bb-e199-42a6-9279-6ff0cdd2c9ec admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'dabc3b39ba454df88179d56dce082443'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:35:43.138241 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-4cd517bb-e199-42a6-9279-6ff0cdd2c9ec admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'dabc3b39ba454df88179d56dce082443'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:35:43.161961 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-383fdf6b-b4f2-4cf7-b708-344fa4f69d3b None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:43.169927 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 827/1644] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:35:42 2022] DELETE /identity/v3/users/dabc3b39ba454df88179d56dce082443 => generated 0 bytes in 398 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:35:43.175622 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c06ad46c-43af-4eb6-b633-aa7635240c11 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:43.176604 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c06ad46c-43af-4eb6-b633-aa7635240c11 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:43.177136 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c06ad46c-43af-4eb6-b633-aa7635240c11 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:43.233010 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-c06ad46c-43af-4eb6-b633-aa7635240c11 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:35:43.235181 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-c06ad46c-43af-4eb6-b633-aa7635240c11 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:35:43.268230 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1279b33f-b9c8-4e53-a7c2-bf7040fd27c9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:43.269225 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1279b33f-b9c8-4e53-a7c2-bf7040fd27c9 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:43.269681 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1279b33f-b9c8-4e53-a7c2-bf7040fd27c9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:43.270416 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1279b33f-b9c8-4e53-a7c2-bf7040fd27c9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:43.272533 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-1279b33f-b9c8-4e53-a7c2-bf7040fd27c9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:35:43.314517 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:35:43.409598 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 828/1645] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:35:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 235 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:35:43.459335 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c06ad46c-43af-4eb6-b633-aa7635240c11 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:43.461007 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-c06ad46c-43af-4eb6-b633-aa7635240c11 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:35:43.463076 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1279b33f-b9c8-4e53-a7c2-bf7040fd27c9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:43.468659 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1279b33f-b9c8-4e53-a7c2-bf7040fd27c9 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:43.486574 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 818/1646] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:35:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 453 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:35:43.492906 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c06ad46c-43af-4eb6-b633-aa7635240c11 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:43.515221 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9773ed94-8d86-454d-b9b7-35a5c3427118 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:43.516725 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9773ed94-8d86-454d-b9b7-35a5c3427118 service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:43.517088 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9773ed94-8d86-454d-b9b7-35a5c3427118 service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:43.517465 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9773ed94-8d86-454d-b9b7-35a5c3427118 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:43.538902 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9773ed94-8d86-454d-b9b7-35a5c3427118 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:43.566405 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9773ed94-8d86-454d-b9b7-35a5c3427118 service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:43.572950 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 829/1647] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:35:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 114 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 25 04:35:45.316188 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1279b33f-b9c8-4e53-a7c2-bf7040fd27c9 service neutron] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:45.317690 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-1279b33f-b9c8-4e53-a7c2-bf7040fd27c9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:35:45.336335 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1279b33f-b9c8-4e53-a7c2-bf7040fd27c9 service neutron] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:45.359508 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71e22944-d120-4380-8c6b-21e7740cc1b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:45.371511 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e22944-d120-4380-8c6b-21e7740cc1b4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:45.371511 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e22944-d120-4380-8c6b-21e7740cc1b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:45.371851 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e22944-d120-4380-8c6b-21e7740cc1b4 admin admin] PATH_INFO: `/v3/projects/8f27e933cfea467784e3621b1d1b7093` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:45.373799 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71e22944-d120-4380-8c6b-21e7740cc1b4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8f27e933cfea467784e3621b1d1b7093)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:45.378527 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71e22944-d120-4380-8c6b-21e7740cc1b4 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:45.492073 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-71e22944-d120-4380-8c6b-21e7740cc1b4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8f27e933cfea467784e3621b1d1b7093', 'request_id': 'req-71e22944-d120-4380-8c6b-21e7740cc1b4'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:35:45.505440 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-71e22944-d120-4380-8c6b-21e7740cc1b4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8f27e933cfea467784e3621b1d1b7093', 'request_id': 'req-71e22944-d120-4380-8c6b-21e7740cc1b4'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:35:45.512774 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9773ed94-8d86-454d-b9b7-35a5c3427118 service neutron] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:45.517117 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-9773ed94-8d86-454d-b9b7-35a5c3427118 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:35:45.546003 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9773ed94-8d86-454d-b9b7-35a5c3427118 service neutron] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:45.547870 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-71e22944-d120-4380-8c6b-21e7740cc1b4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8f27e933cfea467784e3621b1d1b7093', 'request_id': 'req-71e22944-d120-4380-8c6b-21e7740cc1b4'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:35:45.594551 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-71e22944-d120-4380-8c6b-21e7740cc1b4 admin admin] The token cache is being invalidate because project 8f27e933cfea467784e3621b1d1b7093 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:35:45.595000 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-71e22944-d120-4380-8c6b-21e7740cc1b4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:35:45.597480 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-71e22944-d120-4380-8c6b-21e7740cc1b4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:35:45.603962 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 819/1648] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:35:45 2022] DELETE /identity/v3/projects/8f27e933cfea467784e3621b1d1b7093 => generated 0 bytes in 285 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:35:45.605142 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71e22944-d120-4380-8c6b-21e7740cc1b4 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:45.607850 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-71e22944-d120-4380-8c6b-21e7740cc1b4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:35:45.764533 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-011b3c14-51d4-4d3d-875c-60a35606dc7e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:45.765628 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-011b3c14-51d4-4d3d-875c-60a35606dc7e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:45.767389 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-011b3c14-51d4-4d3d-875c-60a35606dc7e admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:45.767748 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-011b3c14-51d4-4d3d-875c-60a35606dc7e admin admin] PATH_INFO: `/v3/projects/b01d6b07a05345deb81df6c18d28a181` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:45.773313 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-011b3c14-51d4-4d3d-875c-60a35606dc7e admin admin] RBAC: Authorizing `identity:delete_project(project_id=b01d6b07a05345deb81df6c18d28a181)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:45.778896 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: WARNING py.warnings [None req-011b3c14-51d4-4d3d-875c-60a35606dc7e admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 25 04:35:45.778896 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: warnings.warn(msg) Sep 25 04:35:45.778896 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: Sep 25 04:35:45.784189 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-011b3c14-51d4-4d3d-875c-60a35606dc7e admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:45.843164 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71e22944-d120-4380-8c6b-21e7740cc1b4 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:45.869552 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f2e0f0c-55d4-4917-98c6-647681368554 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:45.870223 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f2e0f0c-55d4-4917-98c6-647681368554 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:45.870506 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f2e0f0c-55d4-4917-98c6-647681368554 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:45.870784 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f2e0f0c-55d4-4917-98c6-647681368554 admin admin] PATH_INFO: `/v3/users/c9c815de3e9e4e548e092db41154c082` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:45.871988 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f2e0f0c-55d4-4917-98c6-647681368554 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c9c815de3e9e4e548e092db41154c082)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:45.873068 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: WARNING py.warnings [None req-4f2e0f0c-55d4-4917-98c6-647681368554 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 25 04:35:45.873068 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: warnings.warn(msg) Sep 25 04:35:45.873068 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: Sep 25 04:35:45.874836 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f2e0f0c-55d4-4917-98c6-647681368554 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:45.925261 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-011b3c14-51d4-4d3d-875c-60a35606dc7e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b01d6b07a05345deb81df6c18d28a181', 'request_id': 'req-011b3c14-51d4-4d3d-875c-60a35606dc7e'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:35:45.929907 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:35:45.947632 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-011b3c14-51d4-4d3d-875c-60a35606dc7e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b01d6b07a05345deb81df6c18d28a181', 'request_id': 'req-011b3c14-51d4-4d3d-875c-60a35606dc7e'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:35:45.962833 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-011b3c14-51d4-4d3d-875c-60a35606dc7e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b01d6b07a05345deb81df6c18d28a181', 'request_id': 'req-011b3c14-51d4-4d3d-875c-60a35606dc7e'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:35:45.991894 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:35:45.998202 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-011b3c14-51d4-4d3d-875c-60a35606dc7e admin admin] The token cache is being invalidate because project b01d6b07a05345deb81df6c18d28a181 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81434) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:35:46.000571 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-011b3c14-51d4-4d3d-875c-60a35606dc7e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:35:46.002063 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-011b3c14-51d4-4d3d-875c-60a35606dc7e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:35:46.006393 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 830/1649] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:35:45 2022] DELETE /identity/v3/projects/b01d6b07a05345deb81df6c18d28a181 => generated 0 bytes in 497 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 04:35:46.009360 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-4f2e0f0c-55d4-4917-98c6-647681368554 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c9c815de3e9e4e548e092db41154c082'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:35:46.010918 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-011b3c14-51d4-4d3d-875c-60a35606dc7e admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:46.015246 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-011b3c14-51d4-4d3d-875c-60a35606dc7e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:35:46.020510 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-4f2e0f0c-55d4-4917-98c6-647681368554 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c9c815de3e9e4e548e092db41154c082'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:35:46.056233 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-4f2e0f0c-55d4-4917-98c6-647681368554 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c9c815de3e9e4e548e092db41154c082'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:35:46.057120 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-4f2e0f0c-55d4-4917-98c6-647681368554 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c9c815de3e9e4e548e092db41154c082'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:35:46.082963 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-4f2e0f0c-55d4-4917-98c6-647681368554 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c9c815de3e9e4e548e092db41154c082'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:35:46.092933 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 820/1650] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:35:45 2022] DELETE /identity/v3/users/c9c815de3e9e4e548e092db41154c082 => generated 0 bytes in 488 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:35:46.186713 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-011b3c14-51d4-4d3d-875c-60a35606dc7e admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:46.359892 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c83190fd-134f-4733-ae51-fa2f193b7081 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:46.360570 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c83190fd-134f-4733-ae51-fa2f193b7081 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:46.360887 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c83190fd-134f-4733-ae51-fa2f193b7081 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:46.361169 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c83190fd-134f-4733-ae51-fa2f193b7081 admin admin] PATH_INFO: `/v3/users/f0541b72570d4835b4841f98c826ebe8` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:46.368138 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c83190fd-134f-4733-ae51-fa2f193b7081 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f0541b72570d4835b4841f98c826ebe8)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:46.373173 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c83190fd-134f-4733-ae51-fa2f193b7081 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:46.506038 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-c83190fd-134f-4733-ae51-fa2f193b7081 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f0541b72570d4835b4841f98c826ebe8'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:35:46.524543 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-c83190fd-134f-4733-ae51-fa2f193b7081 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f0541b72570d4835b4841f98c826ebe8'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:35:46.556130 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-c83190fd-134f-4733-ae51-fa2f193b7081 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f0541b72570d4835b4841f98c826ebe8'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:35:46.557948 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-c83190fd-134f-4733-ae51-fa2f193b7081 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f0541b72570d4835b4841f98c826ebe8'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:35:46.581110 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-c83190fd-134f-4733-ae51-fa2f193b7081 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f0541b72570d4835b4841f98c826ebe8'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:35:46.597442 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 831/1651] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:35:46 2022] DELETE /identity/v3/users/f0541b72570d4835b4841f98c826ebe8 => generated 0 bytes in 587 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:35:46.755164 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f2e0f0c-55d4-4917-98c6-647681368554 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:46.765120 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-4f2e0f0c-55d4-4917-98c6-647681368554 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:35:46.823316 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f2e0f0c-55d4-4917-98c6-647681368554 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:47.064993 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6709107-68d4-4a7c-9e00-d30f983fda0e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:47.066972 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6709107-68d4-4a7c-9e00-d30f983fda0e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:47.067475 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6709107-68d4-4a7c-9e00-d30f983fda0e admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:47.068307 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6709107-68d4-4a7c-9e00-d30f983fda0e admin admin] PATH_INFO: `/v3/projects/aa02bff12b2f4d82870b443a017b1c18` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:47.069963 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6709107-68d4-4a7c-9e00-d30f983fda0e admin admin] RBAC: Authorizing `identity:delete_project(project_id=aa02bff12b2f4d82870b443a017b1c18)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:47.073871 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6709107-68d4-4a7c-9e00-d30f983fda0e admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:47.255050 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-c6709107-68d4-4a7c-9e00-d30f983fda0e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'aa02bff12b2f4d82870b443a017b1c18', 'request_id': 'req-c6709107-68d4-4a7c-9e00-d30f983fda0e'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:35:47.331894 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c83190fd-134f-4733-ae51-fa2f193b7081 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:47.333509 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-c83190fd-134f-4733-ae51-fa2f193b7081 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:35:47.334665 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-c6709107-68d4-4a7c-9e00-d30f983fda0e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'aa02bff12b2f4d82870b443a017b1c18', 'request_id': 'req-c6709107-68d4-4a7c-9e00-d30f983fda0e'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:35:47.360593 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c83190fd-134f-4733-ae51-fa2f193b7081 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:47.360967 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-c6709107-68d4-4a7c-9e00-d30f983fda0e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'aa02bff12b2f4d82870b443a017b1c18', 'request_id': 'req-c6709107-68d4-4a7c-9e00-d30f983fda0e'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:35:47.444779 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:DELETE=13,keystone:SELECT=86,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:35:47.482681 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-c6709107-68d4-4a7c-9e00-d30f983fda0e admin admin] The token cache is being invalidate because project aa02bff12b2f4d82870b443a017b1c18 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:35:47.483511 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-c6709107-68d4-4a7c-9e00-d30f983fda0e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:35:47.490024 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-c6709107-68d4-4a7c-9e00-d30f983fda0e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:35:47.496194 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 821/1652] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:35:46 2022] DELETE /identity/v3/projects/aa02bff12b2f4d82870b443a017b1c18 => generated 0 bytes in 741 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:35:47.501579 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46381a7d-7d43-4b86-abf9-10e8f6870400 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:47.501579 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46381a7d-7d43-4b86-abf9-10e8f6870400 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:47.505923 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46381a7d-7d43-4b86-abf9-10e8f6870400 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:47.588679 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-46381a7d-7d43-4b86-abf9-10e8f6870400 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:35:47.588679 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-46381a7d-7d43-4b86-abf9-10e8f6870400 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:35:47.630617 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c31e5ef-684d-4a9a-a5bb-551ec1b9cc59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:47.631645 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c31e5ef-684d-4a9a-a5bb-551ec1b9cc59 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:47.632008 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c31e5ef-684d-4a9a-a5bb-551ec1b9cc59 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:47.632362 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c31e5ef-684d-4a9a-a5bb-551ec1b9cc59 admin admin] PATH_INFO: `/v3/projects/c7beb085fe6b41d19fbdf454a0981ff9` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:47.634326 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c31e5ef-684d-4a9a-a5bb-551ec1b9cc59 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c7beb085fe6b41d19fbdf454a0981ff9)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:47.636516 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c31e5ef-684d-4a9a-a5bb-551ec1b9cc59 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:47.760681 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=87,keystone:DELETE=11,keystone:INSERT=2 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:35:47.762227 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-8c31e5ef-684d-4a9a-a5bb-551ec1b9cc59 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c7beb085fe6b41d19fbdf454a0981ff9', 'request_id': 'req-8c31e5ef-684d-4a9a-a5bb-551ec1b9cc59'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:35:47.778699 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-8c31e5ef-684d-4a9a-a5bb-551ec1b9cc59 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c7beb085fe6b41d19fbdf454a0981ff9', 'request_id': 'req-8c31e5ef-684d-4a9a-a5bb-551ec1b9cc59'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:35:47.798950 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 822/1653] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:35:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 299 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:35:47.799294 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-8c31e5ef-684d-4a9a-a5bb-551ec1b9cc59 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c7beb085fe6b41d19fbdf454a0981ff9', 'request_id': 'req-8c31e5ef-684d-4a9a-a5bb-551ec1b9cc59'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:35:47.808901 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5a6b29e-06c2-49f8-9d0b-4463e307d450 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:47.809194 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5a6b29e-06c2-49f8-9d0b-4463e307d450 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:47.809461 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5a6b29e-06c2-49f8-9d0b-4463e307d450 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:47.841722 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-8c31e5ef-684d-4a9a-a5bb-551ec1b9cc59 admin admin] The token cache is being invalidate because project c7beb085fe6b41d19fbdf454a0981ff9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81434) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:35:47.844399 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-8c31e5ef-684d-4a9a-a5bb-551ec1b9cc59 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:35:47.865042 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-8c31e5ef-684d-4a9a-a5bb-551ec1b9cc59 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:35:47.867912 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 832/1654] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:35:47 2022] DELETE /identity/v3/projects/c7beb085fe6b41d19fbdf454a0981ff9 => generated 0 bytes in 537 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 04:35:47.882833 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f8c2e30-56be-4314-bada-808d59dafe48 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:47.883566 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f8c2e30-56be-4314-bada-808d59dafe48 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:47.884066 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f8c2e30-56be-4314-bada-808d59dafe48 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:47.898053 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-a5a6b29e-06c2-49f8-9d0b-4463e307d450 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:35:47.899339 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-a5a6b29e-06c2-49f8-9d0b-4463e307d450 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:35:47.938779 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-0f8c2e30-56be-4314-bada-808d59dafe48 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:35:47.940847 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-0f8c2e30-56be-4314-bada-808d59dafe48 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:35:48.077691 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 823/1655] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:35:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 270 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:35:48.081608 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5a6b29e-06c2-49f8-9d0b-4463e307d450 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:48.104887 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5a6b29e-06c2-49f8-9d0b-4463e307d450 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:48.118845 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 833/1656] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:35:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 239 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:35:48.122377 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-293b45db-fb44-4d55-8a21-e4736785ecdd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:48.123030 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-293b45db-fb44-4d55-8a21-e4736785ecdd admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:48.123488 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-293b45db-fb44-4d55-8a21-e4736785ecdd admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:48.123777 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-293b45db-fb44-4d55-8a21-e4736785ecdd admin admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:48.124324 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da3a715-91f2-43f8-8663-e0348ca3f1cf None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:48.124490 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-293b45db-fb44-4d55-8a21-e4736785ecdd admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:35:48.124689 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da3a715-91f2-43f8-8663-e0348ca3f1cf None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:48.124841 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-293b45db-fb44-4d55-8a21-e4736785ecdd admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:48.124936 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da3a715-91f2-43f8-8663-e0348ca3f1cf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:48.127752 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-293b45db-fb44-4d55-8a21-e4736785ecdd admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:48.164686 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 824/1657] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Sep 25 04:35:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:35:48.172005 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-293b45db-fb44-4d55-8a21-e4736785ecdd admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:48.193076 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-293b45db-fb44-4d55-8a21-e4736785ecdd admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:48.205509 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-5da3a715-91f2-43f8-8663-e0348ca3f1cf None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:35:48.207529 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-5da3a715-91f2-43f8-8663-e0348ca3f1cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:35:48.207795 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ba46497-fd7b-4fe0-9e80-8ff722a321ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:48.208431 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ba46497-fd7b-4fe0-9e80-8ff722a321ba admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:48.208748 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ba46497-fd7b-4fe0-9e80-8ff722a321ba admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:48.209201 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ba46497-fd7b-4fe0-9e80-8ff722a321ba admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:48.209855 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ba46497-fd7b-4fe0-9e80-8ff722a321ba admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:48.211811 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ba46497-fd7b-4fe0-9e80-8ff722a321ba admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:48.235578 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 834/1658] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:35:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:35:48.236290 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5da3a715-91f2-43f8-8663-e0348ca3f1cf None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:48.264130 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5da3a715-91f2-43f8-8663-e0348ca3f1cf None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:48.276214 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-809e946f-9911-4269-825f-c3e66709f364 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:48.276926 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-809e946f-9911-4269-825f-c3e66709f364 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:48.277617 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-809e946f-9911-4269-825f-c3e66709f364 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:48.278233 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-809e946f-9911-4269-825f-c3e66709f364 admin admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:48.280232 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-809e946f-9911-4269-825f-c3e66709f364 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:35:48.280808 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-809e946f-9911-4269-825f-c3e66709f364 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:48.282739 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-809e946f-9911-4269-825f-c3e66709f364 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:48.291733 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 825/1659] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:35:48 2022] POST /identity/v3/projects => generated 387 bytes in 121 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:35:48.297913 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ba46497-fd7b-4fe0-9e80-8ff722a321ba admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:48.328253 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 835/1660] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Sep 25 04:35:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 92 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:35:48.338000 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-809e946f-9911-4269-825f-c3e66709f364 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:48.371503 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-809e946f-9911-4269-825f-c3e66709f364 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:48.413071 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:35:48.508322 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ba46497-fd7b-4fe0-9e80-8ff722a321ba admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:48.526534 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e5de24b-86de-45df-b339-75c0013b6b50 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:48.527566 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e5de24b-86de-45df-b339-75c0013b6b50 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:48.528091 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e5de24b-86de-45df-b339-75c0013b6b50 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:48.528613 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e5de24b-86de-45df-b339-75c0013b6b50 admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:48.529373 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e5de24b-86de-45df-b339-75c0013b6b50 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:48.532286 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e5de24b-86de-45df-b339-75c0013b6b50 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:48.569407 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-412c9400-3e51-4554-8b52-e224ca64959d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:48.570484 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-412c9400-3e51-4554-8b52-e224ca64959d admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:48.574166 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-412c9400-3e51-4554-8b52-e224ca64959d admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:48.574464 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-412c9400-3e51-4554-8b52-e224ca64959d admin admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:48.575167 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-412c9400-3e51-4554-8b52-e224ca64959d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:48.577248 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-412c9400-3e51-4554-8b52-e224ca64959d admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:48.608783 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 826/1661] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:35:48 2022] POST /identity/v3/users => generated 294 bytes in 312 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:35:48.612757 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e5de24b-86de-45df-b339-75c0013b6b50 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:48.646709 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 836/1662] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:35:48 2022] POST /identity/v3/projects => generated 403 bytes in 314 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 04:35:48.650362 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e5de24b-86de-45df-b339-75c0013b6b50 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:48.650582 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-412c9400-3e51-4554-8b52-e224ca64959d admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:48.785026 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:35:48.798459 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-412c9400-3e51-4554-8b52-e224ca64959d admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:48.812380 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34a09b80-7f68-4651-93ad-7d650449c3ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:48.813173 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34a09b80-7f68-4651-93ad-7d650449c3ec admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:48.813467 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34a09b80-7f68-4651-93ad-7d650449c3ec admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:48.813744 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34a09b80-7f68-4651-93ad-7d650449c3ec admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:48.814403 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34a09b80-7f68-4651-93ad-7d650449c3ec admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:48.814797 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f0676e6-3a26-4694-8ae1-244a33faa444 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:48.816366 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34a09b80-7f68-4651-93ad-7d650449c3ec admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:48.817546 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f0676e6-3a26-4694-8ae1-244a33faa444 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:48.817921 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f0676e6-3a26-4694-8ae1-244a33faa444 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:48.818426 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f0676e6-3a26-4694-8ae1-244a33faa444 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:48.820538 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f0676e6-3a26-4694-8ae1-244a33faa444 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:48.825550 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f0676e6-3a26-4694-8ae1-244a33faa444 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:48.863097 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 827/1663] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:35:48 2022] GET /identity/v3/roles => generated 1420 bytes in 251 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 25 04:35:48.867199 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f0676e6-3a26-4694-8ae1-244a33faa444 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:48.877562 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 837/1664] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:35:48 2022] POST /identity/v3/users => generated 302 bytes in 227 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 04:35:48.881335 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34a09b80-7f68-4651-93ad-7d650449c3ec admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:48.892379 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f0676e6-3a26-4694-8ae1-244a33faa444 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:48.898693 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34a09b80-7f68-4651-93ad-7d650449c3ec admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:48.902771 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e660e44-f81c-45f3-a94b-5df3c753b928 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:48.905640 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e660e44-f81c-45f3-a94b-5df3c753b928 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:48.905941 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e660e44-f81c-45f3-a94b-5df3c753b928 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:48.906233 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e660e44-f81c-45f3-a94b-5df3c753b928 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:48.906927 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e660e44-f81c-45f3-a94b-5df3c753b928 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:48.913207 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e660e44-f81c-45f3-a94b-5df3c753b928 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:48.916176 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86a73e71-27bd-4b60-b3d6-f5aa42d3b413 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:48.916850 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a73e71-27bd-4b60-b3d6-f5aa42d3b413 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:48.917139 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a73e71-27bd-4b60-b3d6-f5aa42d3b413 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:48.917429 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a73e71-27bd-4b60-b3d6-f5aa42d3b413 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:48.918089 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86a73e71-27bd-4b60-b3d6-f5aa42d3b413 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:48.919731 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86a73e71-27bd-4b60-b3d6-f5aa42d3b413 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:48.942680 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 828/1665] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:35:48 2022] GET /identity/v3/roles => generated 1420 bytes in 76 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:35:48.946787 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e660e44-f81c-45f3-a94b-5df3c753b928 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:48.949985 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 838/1666] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:35:48 2022] GET /identity/v3/roles => generated 1420 bytes in 66 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 25 04:35:48.951132 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86a73e71-27bd-4b60-b3d6-f5aa42d3b413 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:48.975617 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e660e44-f81c-45f3-a94b-5df3c753b928 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:48.985049 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86a73e71-27bd-4b60-b3d6-f5aa42d3b413 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:48.988915 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e93926b-67b7-4fd0-ac29-b5258ce63e41 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:48.989541 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e93926b-67b7-4fd0-ac29-b5258ce63e41 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:48.989932 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e93926b-67b7-4fd0-ac29-b5258ce63e41 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:48.990219 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e93926b-67b7-4fd0-ac29-b5258ce63e41 admin admin] PATH_INFO: `/v3/projects/413ea30d91d54a339c1514324182c553/users/02ebab903a394f428af84c59b87f8384/roles/8f6cc574056c4278bc9778f14dc4b411` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:48.998741 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2277de5f-7d37-4fc8-aeb0-7f933abdc53f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:48.999383 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2277de5f-7d37-4fc8-aeb0-7f933abdc53f admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:48.999868 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2277de5f-7d37-4fc8-aeb0-7f933abdc53f admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:49.000152 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2277de5f-7d37-4fc8-aeb0-7f933abdc53f admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:49.000837 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2277de5f-7d37-4fc8-aeb0-7f933abdc53f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:49.006264 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2277de5f-7d37-4fc8-aeb0-7f933abdc53f admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:49.028123 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e93926b-67b7-4fd0-ac29-b5258ce63e41 admin admin] RBAC: Authorizing `identity:create_grant(project_id=413ea30d91d54a339c1514324182c553, user_id=02ebab903a394f428af84c59b87f8384, role_id=8f6cc574056c4278bc9778f14dc4b411)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:49.032619 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e93926b-67b7-4fd0-ac29-b5258ce63e41 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:49.034631 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 839/1667] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:35:48 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:35:49.040836 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2277de5f-7d37-4fc8-aeb0-7f933abdc53f admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:49.063296 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2277de5f-7d37-4fc8-aeb0-7f933abdc53f admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:49.093521 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 829/1668] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:35:48 2022] PUT /identity/v3/projects/413ea30d91d54a339c1514324182c553/users/02ebab903a394f428af84c59b87f8384/roles/8f6cc574056c4278bc9778f14dc4b411 => generated 0 bytes in 138 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:35:49.097389 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14a1ee08-86fe-42cf-9ab3-cf0760eed6a2 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:49.097730 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14a1ee08-86fe-42cf-9ab3-cf0760eed6a2 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:49.107287 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14a1ee08-86fe-42cf-9ab3-cf0760eed6a2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:49.241705 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-14a1ee08-86fe-42cf-9ab3-cf0760eed6a2 None None] MFA Rules not processed for user `02ebab903a394f428af84c59b87f8384`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:35:49.242953 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-14a1ee08-86fe-42cf-9ab3-cf0760eed6a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:35:49.282640 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70bb5d22-c245-4b32-9087-96efc812871a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:49.282982 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70bb5d22-c245-4b32-9087-96efc812871a admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:49.283372 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70bb5d22-c245-4b32-9087-96efc812871a admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:49.283778 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70bb5d22-c245-4b32-9087-96efc812871a admin admin] PATH_INFO: `/v3/projects/dde405f4d22940388b59dfee633ae404/users/8ec516bb00234bc69caeeaa111950647/roles/8f6cc574056c4278bc9778f14dc4b411` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:49.319904 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70bb5d22-c245-4b32-9087-96efc812871a admin admin] RBAC: Authorizing `identity:create_grant(project_id=dde405f4d22940388b59dfee633ae404, user_id=8ec516bb00234bc69caeeaa111950647, role_id=8f6cc574056c4278bc9778f14dc4b411)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:49.324985 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70bb5d22-c245-4b32-9087-96efc812871a admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:49.336439 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:35:49.371460 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 840/1669] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:35:49 2022] PUT /identity/v3/projects/dde405f4d22940388b59dfee633ae404/users/8ec516bb00234bc69caeeaa111950647/roles/8f6cc574056c4278bc9778f14dc4b411 => generated 0 bytes in 328 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:35:49.373807 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0165cbdf-402e-4667-92ab-55b3a5f968c3 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:49.374292 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0165cbdf-402e-4667-92ab-55b3a5f968c3 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:49.374895 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0165cbdf-402e-4667-92ab-55b3a5f968c3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:49.393995 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 830/1670] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:35:49 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 298 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 04:35:49.397898 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14a1ee08-86fe-42cf-9ab3-cf0760eed6a2 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:49.417403 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14a1ee08-86fe-42cf-9ab3-cf0760eed6a2 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:49.440447 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-0165cbdf-402e-4667-92ab-55b3a5f968c3 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:35:49.443299 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-0165cbdf-402e-4667-92ab-55b3a5f968c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:35:49.567445 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6fca9b5-07dc-4501-b7a9-6bfcfd9e0680 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:49.568226 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6fca9b5-07dc-4501-b7a9-6bfcfd9e0680 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:49.568516 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6fca9b5-07dc-4501-b7a9-6bfcfd9e0680 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:49.569002 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6fca9b5-07dc-4501-b7a9-6bfcfd9e0680 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:49.570241 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6fca9b5-07dc-4501-b7a9-6bfcfd9e0680 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:49.574087 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6fca9b5-07dc-4501-b7a9-6bfcfd9e0680 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:49.619039 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 841/1671] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:35:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 247 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 04:35:49.652230 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 831/1672] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:35:49 2022] POST /identity/v3/projects => generated 385 bytes in 257 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:35:49.660063 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0165cbdf-402e-4667-92ab-55b3a5f968c3 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:49.665339 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6fca9b5-07dc-4501-b7a9-6bfcfd9e0680 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:49.667444 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-e6fca9b5-07dc-4501-b7a9-6bfcfd9e0680 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:35:49.683093 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0165cbdf-402e-4667-92ab-55b3a5f968c3 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:49.771280 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:35:49.880314 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6fca9b5-07dc-4501-b7a9-6bfcfd9e0680 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:49.889654 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8505a969-233d-4958-9b76-9b796f826ff6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:49.890314 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8505a969-233d-4958-9b76-9b796f826ff6 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:49.890595 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8505a969-233d-4958-9b76-9b796f826ff6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:49.890856 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8505a969-233d-4958-9b76-9b796f826ff6 admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:49.891506 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8505a969-233d-4958-9b76-9b796f826ff6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:49.893596 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8505a969-233d-4958-9b76-9b796f826ff6 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:49.915148 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fcc6021-5c8f-42ce-8576-ec3204aeb5ac service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:49.915996 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fcc6021-5c8f-42ce-8576-ec3204aeb5ac service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:49.916426 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fcc6021-5c8f-42ce-8576-ec3204aeb5ac service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:49.916874 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fcc6021-5c8f-42ce-8576-ec3204aeb5ac service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:49.952547 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fcc6021-5c8f-42ce-8576-ec3204aeb5ac service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:49.954256 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fcc6021-5c8f-42ce-8576-ec3204aeb5ac service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:49.984451 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 842/1673] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:35:49 2022] POST /identity/v3/users => generated 293 bytes in 325 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:35:49.993428 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8505a969-233d-4958-9b76-9b796f826ff6 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:49.995940 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 832/1674] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:35:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 331 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 25 04:35:50.031436 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8505a969-233d-4958-9b76-9b796f826ff6 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:50.052879 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d516e389-9302-413e-8b04-8a4803469871 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:50.053345 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d516e389-9302-413e-8b04-8a4803469871 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:50.053748 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d516e389-9302-413e-8b04-8a4803469871 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:50.054128 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d516e389-9302-413e-8b04-8a4803469871 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:50.055090 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d516e389-9302-413e-8b04-8a4803469871 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:50.057603 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d516e389-9302-413e-8b04-8a4803469871 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:50.088248 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 843/1675] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:35:49 2022] GET /identity/v3/roles => generated 1420 bytes in 95 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:35:50.093393 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fcc6021-5c8f-42ce-8576-ec3204aeb5ac service neutron] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:50.114738 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fcc6021-5c8f-42ce-8576-ec3204aeb5ac service neutron] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:50.127287 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ffdbedd-8a17-45ad-bbf4-9b77cb73737a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:50.127910 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ffdbedd-8a17-45ad-bbf4-9b77cb73737a admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:50.128193 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ffdbedd-8a17-45ad-bbf4-9b77cb73737a admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:50.128471 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ffdbedd-8a17-45ad-bbf4-9b77cb73737a admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:50.129213 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ffdbedd-8a17-45ad-bbf4-9b77cb73737a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:50.130847 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ffdbedd-8a17-45ad-bbf4-9b77cb73737a admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:50.168968 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 833/1676] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:35:50 2022] GET /identity/v3/roles => generated 1420 bytes in 77 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:35:50.173635 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d516e389-9302-413e-8b04-8a4803469871 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:50.190277 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d516e389-9302-413e-8b04-8a4803469871 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:50.203707 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5420dfc-4f59-41de-9247-7a07f667f638 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:50.204414 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5420dfc-4f59-41de-9247-7a07f667f638 admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:50.204720 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5420dfc-4f59-41de-9247-7a07f667f638 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:50.204996 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5420dfc-4f59-41de-9247-7a07f667f638 admin admin] PATH_INFO: `/v3/projects/92acfd3b56304d59bebbda7bb72155bc/users/eeb040f44f3b4312be442d539dab2583/roles/8f6cc574056c4278bc9778f14dc4b411` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:50.241413 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5420dfc-4f59-41de-9247-7a07f667f638 admin admin] RBAC: Authorizing `identity:create_grant(project_id=92acfd3b56304d59bebbda7bb72155bc, user_id=eeb040f44f3b4312be442d539dab2583, role_id=8f6cc574056c4278bc9778f14dc4b411)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:50.244109 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5420dfc-4f59-41de-9247-7a07f667f638 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:50.286844 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 844/1677] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:35:50 2022] PUT /identity/v3/projects/92acfd3b56304d59bebbda7bb72155bc/users/eeb040f44f3b4312be442d539dab2583/roles/8f6cc574056c4278bc9778f14dc4b411 => generated 0 bytes in 113 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:35:50.297646 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb5e594-87f3-4e9b-b0aa-fdd034202871 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:50.297945 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb5e594-87f3-4e9b-b0aa-fdd034202871 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:50.298215 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb5e594-87f3-4e9b-b0aa-fdd034202871 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:50.364701 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-9fb5e594-87f3-4e9b-b0aa-fdd034202871 None None] MFA Rules not processed for user `eeb040f44f3b4312be442d539dab2583`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:35:50.365980 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-9fb5e594-87f3-4e9b-b0aa-fdd034202871 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:35:50.522283 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 834/1678] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:35:50 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 222 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 04:35:50.524162 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5420dfc-4f59-41de-9247-7a07f667f638 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:50.635586 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:INSERT=5,keystone:SELECT=95 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:35:50.669805 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5420dfc-4f59-41de-9247-7a07f667f638 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:50.686868 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ed9013c-a46f-4e70-86c7-ab2a67899ae7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:50.687716 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ed9013c-a46f-4e70-86c7-ab2a67899ae7 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:50.688186 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ed9013c-a46f-4e70-86c7-ab2a67899ae7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:50.688338 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ed9013c-a46f-4e70-86c7-ab2a67899ae7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:50.688961 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ed9013c-a46f-4e70-86c7-ab2a67899ae7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:50.691559 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ed9013c-a46f-4e70-86c7-ab2a67899ae7 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:50.770718 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 845/1679] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:35:50 2022] POST /identity/v3/projects => generated 385 bytes in 248 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:35:50.774454 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fb5e594-87f3-4e9b-b0aa-fdd034202871 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:50.794245 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fb5e594-87f3-4e9b-b0aa-fdd034202871 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:50.972123 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-681ae695-8def-425a-be1f-3d522e20e466 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:50.973076 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-681ae695-8def-425a-be1f-3d522e20e466 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:50.973475 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-681ae695-8def-425a-be1f-3d522e20e466 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:50.973850 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-681ae695-8def-425a-be1f-3d522e20e466 admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:50.975207 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-681ae695-8def-425a-be1f-3d522e20e466 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:50.980064 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-681ae695-8def-425a-be1f-3d522e20e466 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:51.041289 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ed9013c-a46f-4e70-86c7-ab2a67899ae7 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:51.041740 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 835/1680] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:35:50 2022] POST /identity/v3/users => generated 299 bytes in 267 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:35:51.057622 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ed9013c-a46f-4e70-86c7-ab2a67899ae7 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:51.073522 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8704dea5-9b55-409f-9a72-6a74c3f1035c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:51.074566 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8704dea5-9b55-409f-9a72-6a74c3f1035c admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:51.075112 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8704dea5-9b55-409f-9a72-6a74c3f1035c admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:51.077408 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8704dea5-9b55-409f-9a72-6a74c3f1035c admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:51.078160 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8704dea5-9b55-409f-9a72-6a74c3f1035c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:51.084131 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8704dea5-9b55-409f-9a72-6a74c3f1035c admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:51.111644 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 846/1681] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:35:51 2022] GET /identity/v3/roles => generated 1420 bytes in 71 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 25 04:35:51.116432 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-681ae695-8def-425a-be1f-3d522e20e466 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:51.137173 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-681ae695-8def-425a-be1f-3d522e20e466 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:51.152128 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41b7c869-ac99-4995-871d-4454161f91d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:51.153027 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b7c869-ac99-4995-871d-4454161f91d8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:51.154087 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b7c869-ac99-4995-871d-4454161f91d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:51.156230 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b7c869-ac99-4995-871d-4454161f91d8 admin admin] PATH_INFO: `/v3/projects/706b8c4a2f4446aea67f6be7a17d56dd/users/81f27dc3f1df471ca4a49e28d7686461/roles/c51fe7c5ad804040879acfb54269455f` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:51.214924 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41b7c869-ac99-4995-871d-4454161f91d8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=706b8c4a2f4446aea67f6be7a17d56dd, user_id=81f27dc3f1df471ca4a49e28d7686461, role_id=c51fe7c5ad804040879acfb54269455f)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:51.222583 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41b7c869-ac99-4995-871d-4454161f91d8 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:51.255268 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 836/1682] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:35:51 2022] PUT /identity/v3/projects/706b8c4a2f4446aea67f6be7a17d56dd/users/81f27dc3f1df471ca4a49e28d7686461/roles/c51fe7c5ad804040879acfb54269455f => generated 0 bytes in 139 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:35:51.259322 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8704dea5-9b55-409f-9a72-6a74c3f1035c admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:51.276128 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8704dea5-9b55-409f-9a72-6a74c3f1035c admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:51.422175 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-678b4811-6750-483c-b62e-21bdc9c8c64f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:51.425399 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-678b4811-6750-483c-b62e-21bdc9c8c64f admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:51.425866 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-678b4811-6750-483c-b62e-21bdc9c8c64f admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:51.426322 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-678b4811-6750-483c-b62e-21bdc9c8c64f admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:51.427172 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-678b4811-6750-483c-b62e-21bdc9c8c64f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:51.431788 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-678b4811-6750-483c-b62e-21bdc9c8c64f admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:51.467460 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 847/1683] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:35:51 2022] GET /identity/v3/roles => generated 1420 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:35:51.472050 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41b7c869-ac99-4995-871d-4454161f91d8 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:51.483539 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41b7c869-ac99-4995-871d-4454161f91d8 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:51.499408 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98aaee03-ed8e-4824-98d3-17903130b89e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:51.500370 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98aaee03-ed8e-4824-98d3-17903130b89e admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:51.500782 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98aaee03-ed8e-4824-98d3-17903130b89e admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:51.502707 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98aaee03-ed8e-4824-98d3-17903130b89e admin admin] PATH_INFO: `/v3/projects/706b8c4a2f4446aea67f6be7a17d56dd/users/81f27dc3f1df471ca4a49e28d7686461/roles/c51fe7c5ad804040879acfb54269455f` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:51.510814 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98aaee03-ed8e-4824-98d3-17903130b89e admin admin] RBAC: Authorizing `identity:create_grant(project_id=706b8c4a2f4446aea67f6be7a17d56dd, user_id=81f27dc3f1df471ca4a49e28d7686461, role_id=c51fe7c5ad804040879acfb54269455f)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:51.517581 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98aaee03-ed8e-4824-98d3-17903130b89e admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:51.525209 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:35:51.551208 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 837/1684] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:35:51 2022] PUT /identity/v3/projects/706b8c4a2f4446aea67f6be7a17d56dd/users/81f27dc3f1df471ca4a49e28d7686461/roles/c51fe7c5ad804040879acfb54269455f => generated 0 bytes in 79 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:35:51.563782 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e2e222-54bd-445a-9391-cc36414bf3b2 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:51.568122 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e2e222-54bd-445a-9391-cc36414bf3b2 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:51.569202 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e2e222-54bd-445a-9391-cc36414bf3b2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:51.642269 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-81e2e222-54bd-445a-9391-cc36414bf3b2 None None] MFA Rules not processed for user `81f27dc3f1df471ca4a49e28d7686461`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:35:51.646926 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-81e2e222-54bd-445a-9391-cc36414bf3b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:35:51.838963 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 848/1685] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:35:51 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 277 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:35:51.852473 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98aaee03-ed8e-4824-98d3-17903130b89e admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:51.854294 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-98aaee03-ed8e-4824-98d3-17903130b89e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:35:52.012788 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98aaee03-ed8e-4824-98d3-17903130b89e admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:52.020024 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f966f97a-fe8c-4b0a-9182-97740d22e7ef service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ffbc136c77b84e8c87128f7247f4ec03', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:52.021037 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f966f97a-fe8c-4b0a-9182-97740d22e7ef service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:52.021529 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f966f97a-fe8c-4b0a-9182-97740d22e7ef service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:52.022145 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f966f97a-fe8c-4b0a-9182-97740d22e7ef service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:52.039334 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f966f97a-fe8c-4b0a-9182-97740d22e7ef service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:52.042731 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f966f97a-fe8c-4b0a-9182-97740d22e7ef service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:52.064343 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 838/1686] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:35:51 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 213 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:35:53.281966 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81e2e222-54bd-445a-9391-cc36414bf3b2 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:53.284750 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-81e2e222-54bd-445a-9391-cc36414bf3b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:35:53.330183 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81e2e222-54bd-445a-9391-cc36414bf3b2 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:53.334790 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c51feda-7489-4c48-ac4e-4543cb408bf7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ffbc136c77b84e8c87128f7247f4ec03', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:53.337014 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c51feda-7489-4c48-ac4e-4543cb408bf7 service cinder] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:53.343138 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c51feda-7489-4c48-ac4e-4543cb408bf7 service cinder] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:53.343760 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c51feda-7489-4c48-ac4e-4543cb408bf7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:53.439677 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:35:53.462802 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c51feda-7489-4c48-ac4e-4543cb408bf7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:53.464971 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c51feda-7489-4c48-ac4e-4543cb408bf7 service cinder] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:53.498445 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 849/1687] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:35:53 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 217 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:35:53.776997 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c55e86b-cd3b-42b7-9c9a-642f2d491539 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:53.777497 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c55e86b-cd3b-42b7-9c9a-642f2d491539 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:53.777593 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c55e86b-cd3b-42b7-9c9a-642f2d491539 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:53.845802 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-6c55e86b-cd3b-42b7-9c9a-642f2d491539 None None] MFA Rules not processed for user `1d1bf78eb0a94bab884b1df13c081096`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:35:53.847104 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-6c55e86b-cd3b-42b7-9c9a-642f2d491539 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:35:54.058644 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 839/1688] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:35:53 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 282 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:35:54.070365 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aabf2b62-5d1d-4080-b4dd-b6a80abd4fa4 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:54.070671 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aabf2b62-5d1d-4080-b4dd-b6a80abd4fa4 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:54.070921 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aabf2b62-5d1d-4080-b4dd-b6a80abd4fa4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:54.137673 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-aabf2b62-5d1d-4080-b4dd-b6a80abd4fa4 None None] MFA Rules not processed for user `1d1bf78eb0a94bab884b1df13c081096`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:35:54.139629 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-aabf2b62-5d1d-4080-b4dd-b6a80abd4fa4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:35:54.179250 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 850/1689] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:35:54 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:35:54.214489 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3966b20b-b59a-4195-a508-150a69e804ef None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:54.216249 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3966b20b-b59a-4195-a508-150a69e804ef None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:54.216448 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3966b20b-b59a-4195-a508-150a69e804ef None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:54.218363 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 840/1690] 10.222.0.25 () {54 vars in 961 bytes} [Sun Sep 25 04:35:54 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:35:54.253609 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bcadec9-d65a-4299-b5e0-fa1afaf79aba None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:54.254032 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bcadec9-d65a-4299-b5e0-fa1afaf79aba None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:54.254322 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bcadec9-d65a-4299-b5e0-fa1afaf79aba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:54.335967 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-7bcadec9-d65a-4299-b5e0-fa1afaf79aba None None] MFA Rules not processed for user `e703b42c53974daaa608b0ccc3f018b1`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:35:54.338709 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-7bcadec9-d65a-4299-b5e0-fa1afaf79aba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:35:54.352604 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3fd2136-d909-4905-b093-19ba56b8b9eb None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:54.352960 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3fd2136-d909-4905-b093-19ba56b8b9eb None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:54.353449 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3fd2136-d909-4905-b093-19ba56b8b9eb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:54.397880 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-a3fd2136-d909-4905-b093-19ba56b8b9eb None None] MFA Rules not processed for user `2666d4774776498bbdbce539296e76a9`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:35:54.399566 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-a3fd2136-d909-4905-b093-19ba56b8b9eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:35:54.500679 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 851/1691] 10.222.0.25 () {58 vars in 1058 bytes} [Sun Sep 25 04:35:54 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 250 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:35:54.512032 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bcadec9-d65a-4299-b5e0-fa1afaf79aba None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:54.517953 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 841/1692] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:35:54 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 04:35:54.529020 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63ac3111-3e5c-4e09-837a-48cc2f2f44a8 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:54.529312 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63ac3111-3e5c-4e09-837a-48cc2f2f44a8 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:54.530155 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63ac3111-3e5c-4e09-837a-48cc2f2f44a8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:54.533151 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bcadec9-d65a-4299-b5e0-fa1afaf79aba None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:54.542489 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bd8ddde-2505-42b4-8e27-aee93739b915 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e703b42c53974daaa608b0ccc3f018b1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'ResellerAdmin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:54.543227 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bd8ddde-2505-42b4-8e27-aee93739b915 service nova] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:54.543574 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bd8ddde-2505-42b4-8e27-aee93739b915 service nova] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:54.544068 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bd8ddde-2505-42b4-8e27-aee93739b915 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:54.555665 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bd8ddde-2505-42b4-8e27-aee93739b915 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:54.557552 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bd8ddde-2505-42b4-8e27-aee93739b915 service nova] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:54.558137 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-63ac3111-3e5c-4e09-837a-48cc2f2f44a8 None None] MFA Rules not processed for user `2666d4774776498bbdbce539296e76a9`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:35:54.559419 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-63ac3111-3e5c-4e09-837a-48cc2f2f44a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:35:54.581481 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 852/1694] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:35:54 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 70 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 25 04:35:54.584305 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 842/1694] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:35:54 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 45 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:35:54.595980 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d03005c1-f369-49e7-a436-fa315e4d4a2d None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:54.596456 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d03005c1-f369-49e7-a436-fa315e4d4a2d None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:54.596558 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d03005c1-f369-49e7-a436-fa315e4d4a2d None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:54.598543 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72da0d7a-7a16-4b93-bbad-f6d0cbfe6f0e None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:54.598728 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 853/1695] 10.222.0.25 () {54 vars in 961 bytes} [Sun Sep 25 04:35:54 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:35:54.599540 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72da0d7a-7a16-4b93-bbad-f6d0cbfe6f0e None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:54.599923 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72da0d7a-7a16-4b93-bbad-f6d0cbfe6f0e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:54.617005 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e6e316-3fc8-413d-afbb-2e1dcaa0ef8c None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:54.617759 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e6e316-3fc8-413d-afbb-2e1dcaa0ef8c None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:54.618180 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e6e316-3fc8-413d-afbb-2e1dcaa0ef8c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:54.657063 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:INSERT=1,keystone:SELECT=99 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:35:54.667619 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-c0e6e316-3fc8-413d-afbb-2e1dcaa0ef8c None None] MFA Rules not processed for user `e703b42c53974daaa608b0ccc3f018b1`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:35:54.669296 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-c0e6e316-3fc8-413d-afbb-2e1dcaa0ef8c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:35:54.687282 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-72da0d7a-7a16-4b93-bbad-f6d0cbfe6f0e None None] MFA Rules not processed for user `8ec516bb00234bc69caeeaa111950647`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:35:54.691836 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-72da0d7a-7a16-4b93-bbad-f6d0cbfe6f0e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:35:54.700196 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 854/1696] 10.222.0.25 () {58 vars in 1058 bytes} [Sun Sep 25 04:35:54 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:35:54.708219 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0e6e316-3fc8-413d-afbb-2e1dcaa0ef8c None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:54.724206 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0e6e316-3fc8-413d-afbb-2e1dcaa0ef8c None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:54.743798 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52e57897-d81a-4d87-b770-a9f540e11575 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e703b42c53974daaa608b0ccc3f018b1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'ResellerAdmin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:54.744433 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52e57897-d81a-4d87-b770-a9f540e11575 service nova] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:54.744770 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52e57897-d81a-4d87-b770-a9f540e11575 service nova] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:54.745351 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52e57897-d81a-4d87-b770-a9f540e11575 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:54.763065 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52e57897-d81a-4d87-b770-a9f540e11575 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:54.764871 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52e57897-d81a-4d87-b770-a9f540e11575 service nova] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:54.781566 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 855/1697] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:35:54 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 74 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 25 04:35:54.820907 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 843/1698] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:35:54 2022] POST /identity/v3/auth/tokens => generated 3280 bytes in 223 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 04:35:54.825212 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52e57897-d81a-4d87-b770-a9f540e11575 service nova] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:54.917282 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52e57897-d81a-4d87-b770-a9f540e11575 service nova] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:54.929511 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c6840ba-80a8-4dd4-b07c-df6b0e36bac0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:54.930153 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c6840ba-80a8-4dd4-b07c-df6b0e36bac0 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:54.930438 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c6840ba-80a8-4dd4-b07c-df6b0e36bac0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:54.930712 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c6840ba-80a8-4dd4-b07c-df6b0e36bac0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:54.931356 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c6840ba-80a8-4dd4-b07c-df6b0e36bac0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:54.935790 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c6840ba-80a8-4dd4-b07c-df6b0e36bac0 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:54.959080 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22a68c87-2dad-4ad4-ac80-9967e0d93e11 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:54.959368 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22a68c87-2dad-4ad4-ac80-9967e0d93e11 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:54.959643 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22a68c87-2dad-4ad4-ac80-9967e0d93e11 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:54.961376 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 844/1699] 10.222.0.25 () {54 vars in 912 bytes} [Sun Sep 25 04:35:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:35:54.973798 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8871c662-fbff-412b-ac6a-fc0203961086 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:54.974082 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8871c662-fbff-412b-ac6a-fc0203961086 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:54.974408 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8871c662-fbff-412b-ac6a-fc0203961086 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:55.024220 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 856/1700] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:35:54 2022] POST /identity/v3/projects => generated 401 bytes in 200 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:35:55.025925 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-8871c662-fbff-412b-ac6a-fc0203961086 None None] MFA Rules not processed for user `e703b42c53974daaa608b0ccc3f018b1`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:35:55.027716 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-371061f6-8192-466c-9cb4-4fa476c0d1af None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:55.029147 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-371061f6-8192-466c-9cb4-4fa476c0d1af None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:55.029978 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-8871c662-fbff-412b-ac6a-fc0203961086 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:35:55.030296 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-371061f6-8192-466c-9cb4-4fa476c0d1af None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:55.064890 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:35:55.102300 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-371061f6-8192-466c-9cb4-4fa476c0d1af None None] MFA Rules not processed for user `c50ba6f9a164463eae9a248936e7bf80`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:35:55.103744 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-371061f6-8192-466c-9cb4-4fa476c0d1af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:35:55.206496 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 845/1701] 10.222.0.25 () {58 vars in 1009 bytes} [Sun Sep 25 04:35:54 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 233 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 04:35:55.209558 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8871c662-fbff-412b-ac6a-fc0203961086 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:55.295855 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 857/1702] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:35:55 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 271 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:35:55.313425 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-371061f6-8192-466c-9cb4-4fa476c0d1af None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:55.317759 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-371061f6-8192-466c-9cb4-4fa476c0d1af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:35:55.447975 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8871c662-fbff-412b-ac6a-fc0203961086 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:55.464184 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5d4bcce-7b87-46ef-a1f4-074c2fe5f6e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:55.465830 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5d4bcce-7b87-46ef-a1f4-074c2fe5f6e5 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:55.466290 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5d4bcce-7b87-46ef-a1f4-074c2fe5f6e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:55.466571 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5d4bcce-7b87-46ef-a1f4-074c2fe5f6e5 admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:55.469913 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5d4bcce-7b87-46ef-a1f4-074c2fe5f6e5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:55.488407 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5d4bcce-7b87-46ef-a1f4-074c2fe5f6e5 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:55.545926 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 846/1703] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:35:55 2022] POST /identity/v3/users => generated 307 bytes in 335 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:35:55.545926 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30e18559-eb55-4f75-98ef-22e8df86ce72 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:55.545926 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30e18559-eb55-4f75-98ef-22e8df86ce72 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:55.546218 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30e18559-eb55-4f75-98ef-22e8df86ce72 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:55.549175 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 847/1704] 10.222.0.25 () {54 vars in 912 bytes} [Sun Sep 25 04:35:55 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:35:55.550097 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30e18559-eb55-4f75-98ef-22e8df86ce72 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:55.573163 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30e18559-eb55-4f75-98ef-22e8df86ce72 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:55.579441 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f69e3ad-85d8-440d-a708-25c5f6152db0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:55.580077 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f69e3ad-85d8-440d-a708-25c5f6152db0 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:55.580354 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f69e3ad-85d8-440d-a708-25c5f6152db0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:55.580681 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f69e3ad-85d8-440d-a708-25c5f6152db0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:55.581438 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f69e3ad-85d8-440d-a708-25c5f6152db0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:55.584148 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f69e3ad-85d8-440d-a708-25c5f6152db0 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:55.596772 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-371061f6-8192-466c-9cb4-4fa476c0d1af None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:55.608406 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e00ef8be-8a36-454d-80e3-0c8b307ac6dc service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '316eaa67d5464091af47efd489b96b87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:55.609607 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e00ef8be-8a36-454d-80e3-0c8b307ac6dc service glance] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:55.610809 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e00ef8be-8a36-454d-80e3-0c8b307ac6dc service glance] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:55.611163 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e00ef8be-8a36-454d-80e3-0c8b307ac6dc service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:55.622402 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 848/1705] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:35:55 2022] GET /identity/v3/roles => generated 1420 bytes in 70 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:35:55.623256 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e48ad55-dbac-41ac-a31c-e03c282bae14 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:55.623600 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e48ad55-dbac-41ac-a31c-e03c282bae14 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:55.624077 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e48ad55-dbac-41ac-a31c-e03c282bae14 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:55.640530 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e00ef8be-8a36-454d-80e3-0c8b307ac6dc service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:55.644700 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e00ef8be-8a36-454d-80e3-0c8b307ac6dc service glance] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:55.667850 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-1e48ad55-dbac-41ac-a31c-e03c282bae14 None None] MFA Rules not processed for user `e703b42c53974daaa608b0ccc3f018b1`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:35:55.669529 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-1e48ad55-dbac-41ac-a31c-e03c282bae14 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:35:55.672930 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 858/1706] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:35:55 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 377 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:35:55.673727 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e00ef8be-8a36-454d-80e3-0c8b307ac6dc service glance] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:55.699283 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e00ef8be-8a36-454d-80e3-0c8b307ac6dc service glance] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:55.708716 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 849/1707] 10.222.0.25 () {58 vars in 1009 bytes} [Sun Sep 25 04:35:55 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:35:55.711685 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0372b783-aa7a-4b8e-b489-f2f1776091e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:55.713168 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0372b783-aa7a-4b8e-b489-f2f1776091e7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:55.714991 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0372b783-aa7a-4b8e-b489-f2f1776091e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:55.715178 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0372b783-aa7a-4b8e-b489-f2f1776091e7 admin admin] PATH_INFO: `/v3/projects/5ae19fc4d2ec4d4c979274be0bd3d14a/users/2ca34666ae954013b68c5a931898940d/roles/c51fe7c5ad804040879acfb54269455f` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:55.722890 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e48ad55-dbac-41ac-a31c-e03c282bae14 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:55.724453 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-1e48ad55-dbac-41ac-a31c-e03c282bae14 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:35:55.733270 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:35:55.742492 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e48ad55-dbac-41ac-a31c-e03c282bae14 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:55.757486 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0372b783-aa7a-4b8e-b489-f2f1776091e7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5ae19fc4d2ec4d4c979274be0bd3d14a, user_id=2ca34666ae954013b68c5a931898940d, role_id=c51fe7c5ad804040879acfb54269455f)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:55.761934 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0372b783-aa7a-4b8e-b489-f2f1776091e7 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:55.778421 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e11c1d14-b480-4a40-a674-60961bce8649 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '316eaa67d5464091af47efd489b96b87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:55.780864 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e11c1d14-b480-4a40-a674-60961bce8649 service glance] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:55.781352 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e11c1d14-b480-4a40-a674-60961bce8649 service glance] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:55.782519 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e11c1d14-b480-4a40-a674-60961bce8649 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:55.801619 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 859/1708] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:35:55 2022] PUT /identity/v3/projects/5ae19fc4d2ec4d4c979274be0bd3d14a/users/2ca34666ae954013b68c5a931898940d/roles/c51fe7c5ad804040879acfb54269455f => generated 0 bytes in 125 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:35:55.802876 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0372b783-aa7a-4b8e-b489-f2f1776091e7 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:55.807356 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e11c1d14-b480-4a40-a674-60961bce8649 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:55.809285 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e11c1d14-b480-4a40-a674-60961bce8649 service glance] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:55.846273 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 850/1709] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:35:55 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 124 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 25 04:35:55.981090 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:35:55.994157 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0372b783-aa7a-4b8e-b489-f2f1776091e7 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:56.007113 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80fcd9c7-4622-4aec-a6b9-4f9ca6c263c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:56.007928 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80fcd9c7-4622-4aec-a6b9-4f9ca6c263c0 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:56.008444 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80fcd9c7-4622-4aec-a6b9-4f9ca6c263c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:56.008915 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80fcd9c7-4622-4aec-a6b9-4f9ca6c263c0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:56.009678 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80fcd9c7-4622-4aec-a6b9-4f9ca6c263c0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:56.011403 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80fcd9c7-4622-4aec-a6b9-4f9ca6c263c0 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:56.027727 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 860/1710] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:35:55 2022] GET /identity/v3/roles => generated 1420 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:35:56.032009 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e11c1d14-b480-4a40-a674-60961bce8649 service glance] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:56.048142 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e11c1d14-b480-4a40-a674-60961bce8649 service glance] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:56.059514 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad7e6bfd-fc4c-437d-b379-648799fce3b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:56.060521 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad7e6bfd-fc4c-437d-b379-648799fce3b7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:56.061851 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad7e6bfd-fc4c-437d-b379-648799fce3b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:56.062352 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad7e6bfd-fc4c-437d-b379-648799fce3b7 admin admin] PATH_INFO: `/v3/projects/5ae19fc4d2ec4d4c979274be0bd3d14a/users/2ca34666ae954013b68c5a931898940d/roles/c51fe7c5ad804040879acfb54269455f` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:56.064695 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad7e6bfd-fc4c-437d-b379-648799fce3b7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5ae19fc4d2ec4d4c979274be0bd3d14a, user_id=2ca34666ae954013b68c5a931898940d, role_id=c51fe7c5ad804040879acfb54269455f)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:56.068286 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad7e6bfd-fc4c-437d-b379-648799fce3b7 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:56.087609 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 851/1711] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:35:56 2022] PUT /identity/v3/projects/5ae19fc4d2ec4d4c979274be0bd3d14a/users/2ca34666ae954013b68c5a931898940d/roles/c51fe7c5ad804040879acfb54269455f => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 04:35:56.551848 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbd0e533-55c4-41de-8705-a0e14fa47a1a None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:56.552253 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbd0e533-55c4-41de-8705-a0e14fa47a1a None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:56.552654 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbd0e533-55c4-41de-8705-a0e14fa47a1a None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:56.554394 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 861/1712] 10.222.0.25 () {54 vars in 912 bytes} [Sun Sep 25 04:35:56 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:35:56.578721 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e372d26-d630-4ff7-9c0a-10e709e29ea2 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:56.580945 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e372d26-d630-4ff7-9c0a-10e709e29ea2 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:56.581344 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e372d26-d630-4ff7-9c0a-10e709e29ea2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:56.665406 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-4e372d26-d630-4ff7-9c0a-10e709e29ea2 None None] MFA Rules not processed for user `0268e88e0c5645a98bab5664be6e49bc`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:35:56.667147 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-4e372d26-d630-4ff7-9c0a-10e709e29ea2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:35:56.680749 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-623e9163-5f04-44c2-81f9-440bb55a5539 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:56.681827 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-623e9163-5f04-44c2-81f9-440bb55a5539 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:56.682314 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-623e9163-5f04-44c2-81f9-440bb55a5539 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:56.684787 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 862/1713] 10.222.0.25 () {54 vars in 912 bytes} [Sun Sep 25 04:35:56 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 25 04:35:56.696775 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3fe2494-d8ad-47a1-9f3d-c25b7d60efb1 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:56.697289 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3fe2494-d8ad-47a1-9f3d-c25b7d60efb1 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:56.697771 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3fe2494-d8ad-47a1-9f3d-c25b7d60efb1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:56.733008 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-d3fe2494-d8ad-47a1-9f3d-c25b7d60efb1 None None] MFA Rules not processed for user `0268e88e0c5645a98bab5664be6e49bc`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:35:56.736492 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-d3fe2494-d8ad-47a1-9f3d-c25b7d60efb1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:35:56.816996 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 852/1714] 10.222.0.25 () {58 vars in 1009 bytes} [Sun Sep 25 04:35:56 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 236 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:35:56.824958 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e372d26-d630-4ff7-9c0a-10e709e29ea2 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:56.828999 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-4e372d26-d630-4ff7-9c0a-10e709e29ea2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:35:56.848045 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e372d26-d630-4ff7-9c0a-10e709e29ea2 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:56.857266 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26285dde-bfac-4499-90dc-25769c450771 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:56.859629 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26285dde-bfac-4499-90dc-25769c450771 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:56.861685 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26285dde-bfac-4499-90dc-25769c450771 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:56.863103 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26285dde-bfac-4499-90dc-25769c450771 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:56.874415 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 863/1715] 10.222.0.25 () {58 vars in 1009 bytes} [Sun Sep 25 04:35:56 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:35:56.882399 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3fe2494-d8ad-47a1-9f3d-c25b7d60efb1 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:56.887406 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26285dde-bfac-4499-90dc-25769c450771 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:56.893489 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26285dde-bfac-4499-90dc-25769c450771 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:56.895823 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3fe2494-d8ad-47a1-9f3d-c25b7d60efb1 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:56.909038 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4902155f-aca3-4121-b8ba-951cb79dd024 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:56.909855 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4902155f-aca3-4121-b8ba-951cb79dd024 service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:56.911280 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4902155f-aca3-4121-b8ba-951cb79dd024 service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:56.911673 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4902155f-aca3-4121-b8ba-951cb79dd024 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:56.917436 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 853/1716] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:35:56 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:35:56.925390 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4902155f-aca3-4121-b8ba-951cb79dd024 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:56.927311 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4902155f-aca3-4121-b8ba-951cb79dd024 service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:56.937808 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 864/1717] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:35:56 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 56 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 25 04:35:57.786667 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26285dde-bfac-4499-90dc-25769c450771 service neutron] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:57.926744 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4902155f-aca3-4121-b8ba-951cb79dd024 service neutron] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:35:57.928253 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-4902155f-aca3-4121-b8ba-951cb79dd024 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:35:57.986513 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26285dde-bfac-4499-90dc-25769c450771 service neutron] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:58.004660 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82e3a0ae-20a7-41c0-94b1-49b85c236375 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e703b42c53974daaa608b0ccc3f018b1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'ResellerAdmin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:58.005651 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82e3a0ae-20a7-41c0-94b1-49b85c236375 service nova] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:58.006076 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82e3a0ae-20a7-41c0-94b1-49b85c236375 service nova] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:58.006489 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82e3a0ae-20a7-41c0-94b1-49b85c236375 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:58.106846 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4902155f-aca3-4121-b8ba-951cb79dd024 service neutron] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:35:58.120053 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8d92274-8b87-4a40-a40c-dfc3bb664c94 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '676d04ef5a984edcaad63d9156568660', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:35:58.122249 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8d92274-8b87-4a40-a40c-dfc3bb664c94 service placement] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:58.122533 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8d92274-8b87-4a40-a40c-dfc3bb664c94 service placement] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:58.122838 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8d92274-8b87-4a40-a40c-dfc3bb664c94 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:58.147430 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-a8d92274-8b87-4a40-a40c-dfc3bb664c94 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:35:58.166254 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:35:58.173522 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82e3a0ae-20a7-41c0-94b1-49b85c236375 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:58.176913 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82e3a0ae-20a7-41c0-94b1-49b85c236375 service nova] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:58.193603 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8d92274-8b87-4a40-a40c-dfc3bb664c94 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:35:58.199399 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8d92274-8b87-4a40-a40c-dfc3bb664c94 service placement] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:35:58.218824 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 854/1718] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:35:57 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 432 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:35:58.281119 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 865/1719] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:35:57 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 355 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:35:58.868493 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb52efb-0604-4fc2-b162-ed52ddf646f5 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:58.869050 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb52efb-0604-4fc2-b162-ed52ddf646f5 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:58.869419 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb52efb-0604-4fc2-b162-ed52ddf646f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:35:58.904163 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-1fb52efb-0604-4fc2-b162-ed52ddf646f5 None None] MFA Rules not processed for user `2666d4774776498bbdbce539296e76a9`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:35:58.905724 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-1fb52efb-0604-4fc2-b162-ed52ddf646f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:35:59.067571 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 855/1720] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:35:58 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 200 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:35:59.962353 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b99be69-b024-4f12-828a-ea81e2f0d5b3 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:35:59.963021 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b99be69-b024-4f12-828a-ea81e2f0d5b3 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:35:59.963021 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b99be69-b024-4f12-828a-ea81e2f0d5b3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:00.062362 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-7b99be69-b024-4f12-828a-ea81e2f0d5b3 None None] MFA Rules not processed for user `2ca34666ae954013b68c5a931898940d`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:36:00.063629 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-7b99be69-b024-4f12-828a-ea81e2f0d5b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:00.215189 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fb52efb-0604-4fc2-b162-ed52ddf646f5 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:00.282506 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 866/1721] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:35:59 2022] POST /identity/v3/auth/tokens => generated 3345 bytes in 321 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:36:00.370234 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fb52efb-0604-4fc2-b162-ed52ddf646f5 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:00.379713 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80ce40da-60dc-4593-bb5d-fc1813a2d2d3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ffbc136c77b84e8c87128f7247f4ec03', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:00.383010 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80ce40da-60dc-4593-bb5d-fc1813a2d2d3 service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:00.383645 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80ce40da-60dc-4593-bb5d-fc1813a2d2d3 service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:00.384875 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80ce40da-60dc-4593-bb5d-fc1813a2d2d3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:00.614281 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80ce40da-60dc-4593-bb5d-fc1813a2d2d3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:00.621579 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80ce40da-60dc-4593-bb5d-fc1813a2d2d3 service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:00.673838 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 856/1722] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:36:00 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 460 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 25 04:36:02.474975 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4da11e5-323d-4707-913b-3635e1717462 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:02.476143 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4da11e5-323d-4707-913b-3635e1717462 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:02.476865 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4da11e5-323d-4707-913b-3635e1717462 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:02.478992 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 867/1723] 10.222.0.25 () {54 vars in 921 bytes} [Sun Sep 25 04:36:02 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:36:02.500156 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e290f501-57be-4f7f-94e4-2a2084dc82e8 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:02.501903 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e290f501-57be-4f7f-94e4-2a2084dc82e8 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:02.502500 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e290f501-57be-4f7f-94e4-2a2084dc82e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:02.538876 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-e290f501-57be-4f7f-94e4-2a2084dc82e8 None None] MFA Rules not processed for user `676d04ef5a984edcaad63d9156568660`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:36:02.547587 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-e290f501-57be-4f7f-94e4-2a2084dc82e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:02.578258 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 857/1724] 10.222.0.25 () {58 vars in 1018 bytes} [Sun Sep 25 04:36:02 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:36:02.597736 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10e3e3c2-2eda-4165-9c98-980e9111912d None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:02.598275 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10e3e3c2-2eda-4165-9c98-980e9111912d None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:02.598698 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10e3e3c2-2eda-4165-9c98-980e9111912d None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:02.600606 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 868/1725] 10.222.0.25 () {54 vars in 961 bytes} [Sun Sep 25 04:36:02 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:36:02.626731 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d50e7f47-5132-4e22-be58-a3f8a633e4f0 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:02.627254 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d50e7f47-5132-4e22-be58-a3f8a633e4f0 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:02.627372 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d50e7f47-5132-4e22-be58-a3f8a633e4f0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:02.699342 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-d50e7f47-5132-4e22-be58-a3f8a633e4f0 None None] MFA Rules not processed for user `676d04ef5a984edcaad63d9156568660`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:36:02.713046 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-d50e7f47-5132-4e22-be58-a3f8a633e4f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:02.753407 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10e3e3c2-2eda-4165-9c98-980e9111912d None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:02.762715 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:36:02.770471 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 858/1726] 10.222.0.25 () {58 vars in 1058 bytes} [Sun Sep 25 04:36:02 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:36:02.798361 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10e3e3c2-2eda-4165-9c98-980e9111912d None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:02.818365 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1482531-735a-49f9-a52c-a2b2a974d50c service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '676d04ef5a984edcaad63d9156568660', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:02.818942 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1482531-735a-49f9-a52c-a2b2a974d50c service placement] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:02.819477 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1482531-735a-49f9-a52c-a2b2a974d50c service placement] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:02.819684 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1482531-735a-49f9-a52c-a2b2a974d50c service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:02.850218 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1482531-735a-49f9-a52c-a2b2a974d50c service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:02.854415 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1482531-735a-49f9-a52c-a2b2a974d50c service placement] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:02.859826 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0a57a1b-99e6-4afc-b533-14c25b8c7753 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:02.860112 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0a57a1b-99e6-4afc-b533-14c25b8c7753 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:02.860668 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0a57a1b-99e6-4afc-b533-14c25b8c7753 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:02.916928 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 869/1727] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:36:02 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 163 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 25 04:36:03.053300 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-b0a57a1b-99e6-4afc-b533-14c25b8c7753 None None] MFA Rules not processed for user `050b3aab44bb4b17aa4a1ba7cccdb231`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:36:03.054577 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-b0a57a1b-99e6-4afc-b533-14c25b8c7753 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:03.093910 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8086395-9bee-413f-b764-47ce83e7784c None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:03.094442 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8086395-9bee-413f-b764-47ce83e7784c None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:03.094871 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8086395-9bee-413f-b764-47ce83e7784c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:03.119782 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:36:03.154870 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-b8086395-9bee-413f-b764-47ce83e7784c None None] MFA Rules not processed for user `e7354aa144df4ad69cd7d12f11b1d8a5`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:36:03.158011 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-b8086395-9bee-413f-b764-47ce83e7784c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:03.253277 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 859/1728] 10.222.0.25 () {58 vars in 1009 bytes} [Sun Sep 25 04:36:02 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 394 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 04:36:03.306477 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 870/1729] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:36:03 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 214 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:36:03.316296 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0a57a1b-99e6-4afc-b533-14c25b8c7753 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:03.336216 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0a57a1b-99e6-4afc-b533-14c25b8c7753 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:03.344612 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3e7e065-96dc-4eac-9bab-de0fa4e3f7d9 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e703b42c53974daaa608b0ccc3f018b1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'ResellerAdmin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:03.345609 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e7e065-96dc-4eac-9bab-de0fa4e3f7d9 service nova] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:03.346376 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e7e065-96dc-4eac-9bab-de0fa4e3f7d9 service nova] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:03.347224 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e7e065-96dc-4eac-9bab-de0fa4e3f7d9 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:03.367203 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3e7e065-96dc-4eac-9bab-de0fa4e3f7d9 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:03.369623 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3e7e065-96dc-4eac-9bab-de0fa4e3f7d9 service nova] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:03.381649 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 860/1730] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:36:03 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 66 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:36:03.561484 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8086395-9bee-413f-b764-47ce83e7784c None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:03.563206 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-b8086395-9bee-413f-b764-47ce83e7784c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:03.814335 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8086395-9bee-413f-b764-47ce83e7784c None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:03.837569 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5052738e-353b-4d08-8f30-49a6a4e0d807 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '23ea88f46e7740d295148a93ade29dd9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:03.838184 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5052738e-353b-4d08-8f30-49a6a4e0d807 service swift] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:03.838466 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5052738e-353b-4d08-8f30-49a6a4e0d807 service swift] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:03.838743 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5052738e-353b-4d08-8f30-49a6a4e0d807 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:03.861958 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5052738e-353b-4d08-8f30-49a6a4e0d807 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:03.863861 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5052738e-353b-4d08-8f30-49a6a4e0d807 service swift] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:03.874776 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 871/1731] 10.222.0.25 () {60 vars in 1384 bytes} [Sun Sep 25 04:36:03 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 25 04:36:05.371516 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3e7e065-96dc-4eac-9bab-de0fa4e3f7d9 service nova] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:05.556896 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3e7e065-96dc-4eac-9bab-de0fa4e3f7d9 service nova] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:05.587957 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0ffa048-33bc-401b-b71b-514a2390a153 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:05.593292 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0ffa048-33bc-401b-b71b-514a2390a153 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:05.595029 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0ffa048-33bc-401b-b71b-514a2390a153 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:05.595702 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0ffa048-33bc-401b-b71b-514a2390a153 admin admin] PATH_INFO: `/v3/users/02ebab903a394f428af84c59b87f8384` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:05.599802 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0ffa048-33bc-401b-b71b-514a2390a153 admin admin] RBAC: Authorizing `identity:delete_user(user_id=02ebab903a394f428af84c59b87f8384)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:05.606601 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0ffa048-33bc-401b-b71b-514a2390a153 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:05.767789 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-a0ffa048-33bc-401b-b71b-514a2390a153 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '02ebab903a394f428af84c59b87f8384'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:05.781302 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-a0ffa048-33bc-401b-b71b-514a2390a153 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '02ebab903a394f428af84c59b87f8384'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:05.825007 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-a0ffa048-33bc-401b-b71b-514a2390a153 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '02ebab903a394f428af84c59b87f8384'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:05.829148 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-a0ffa048-33bc-401b-b71b-514a2390a153 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '02ebab903a394f428af84c59b87f8384'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:05.848189 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-a0ffa048-33bc-401b-b71b-514a2390a153 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '02ebab903a394f428af84c59b87f8384'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:05.872331 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 861/1732] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:36:05 2022] DELETE /identity/v3/users/02ebab903a394f428af84c59b87f8384 => generated 0 bytes in 499 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:36:05.874086 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bbcd72e-f92f-444d-875c-a00965ee8aa9 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:05.874616 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bbcd72e-f92f-444d-875c-a00965ee8aa9 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:05.874616 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bbcd72e-f92f-444d-875c-a00965ee8aa9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:05.944975 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-1bbcd72e-f92f-444d-875c-a00965ee8aa9 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:36:05.959029 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-1bbcd72e-f92f-444d-875c-a00965ee8aa9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:06.234784 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:36:06.335373 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 872/1733] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:36:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 462 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:36:06.461352 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0ffa048-33bc-401b-b71b-514a2390a153 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:06.463716 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-a0ffa048-33bc-401b-b71b-514a2390a153 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:06.537760 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:36:06.603698 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0ffa048-33bc-401b-b71b-514a2390a153 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:06.610529 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc34d5dd-9323-4940-9bf6-2baac7eea43d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:06.612023 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc34d5dd-9323-4940-9bf6-2baac7eea43d service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:06.613104 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc34d5dd-9323-4940-9bf6-2baac7eea43d service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:06.613621 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc34d5dd-9323-4940-9bf6-2baac7eea43d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:06.631291 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc34d5dd-9323-4940-9bf6-2baac7eea43d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:06.659710 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc34d5dd-9323-4940-9bf6-2baac7eea43d service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:06.676468 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 862/1734] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:36:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 216 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 25 04:36:08.012327 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-819b1cd2-59dd-4198-a096-cf788a8d3297 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:08.012944 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-819b1cd2-59dd-4198-a096-cf788a8d3297 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:08.012944 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-819b1cd2-59dd-4198-a096-cf788a8d3297 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:08.014326 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 873/1735] 10.222.0.25 () {54 vars in 921 bytes} [Sun Sep 25 04:36:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:36:08.191812 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bc3a45f-d3c4-4cff-8bb1-de0258630d02 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:08.192388 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bc3a45f-d3c4-4cff-8bb1-de0258630d02 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:08.192388 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bc3a45f-d3c4-4cff-8bb1-de0258630d02 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:08.242528 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-2bc3a45f-d3c4-4cff-8bb1-de0258630d02 None None] MFA Rules not processed for user `676d04ef5a984edcaad63d9156568660`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:36:08.246448 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-2bc3a45f-d3c4-4cff-8bb1-de0258630d02 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:08.501018 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 863/1736] 10.222.0.25 () {58 vars in 1018 bytes} [Sun Sep 25 04:36:08 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 311 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:36:08.651972 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-819b1cd2-59dd-4198-a096-cf788a8d3297 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:08.654681 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-819b1cd2-59dd-4198-a096-cf788a8d3297 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:08.690324 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-819b1cd2-59dd-4198-a096-cf788a8d3297 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:08.703472 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34174ce1-9b4e-4af7-adb9-f8bb05744b31 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:08.711730 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34174ce1-9b4e-4af7-adb9-f8bb05744b31 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:08.711730 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34174ce1-9b4e-4af7-adb9-f8bb05744b31 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:08.711730 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34174ce1-9b4e-4af7-adb9-f8bb05744b31 admin admin] PATH_INFO: `/v3/users/8ec516bb00234bc69caeeaa111950647` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:08.711730 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34174ce1-9b4e-4af7-adb9-f8bb05744b31 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8ec516bb00234bc69caeeaa111950647)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:08.715633 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34174ce1-9b4e-4af7-adb9-f8bb05744b31 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:08.804842 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bc3a45f-d3c4-4cff-8bb1-de0258630d02 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:08.806495 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-34174ce1-9b4e-4af7-adb9-f8bb05744b31 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8ec516bb00234bc69caeeaa111950647'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:08.806679 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-2bc3a45f-d3c4-4cff-8bb1-de0258630d02 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:08.824566 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-34174ce1-9b4e-4af7-adb9-f8bb05744b31 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8ec516bb00234bc69caeeaa111950647'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:08.846685 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-34174ce1-9b4e-4af7-adb9-f8bb05744b31 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8ec516bb00234bc69caeeaa111950647'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:08.847807 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-34174ce1-9b4e-4af7-adb9-f8bb05744b31 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8ec516bb00234bc69caeeaa111950647'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:08.847945 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bc3a45f-d3c4-4cff-8bb1-de0258630d02 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:08.860407 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-34174ce1-9b4e-4af7-adb9-f8bb05744b31 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8ec516bb00234bc69caeeaa111950647'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:08.871448 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 874/1737] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:36:08 2022] DELETE /identity/v3/users/8ec516bb00234bc69caeeaa111950647 => generated 0 bytes in 220 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:36:08.877986 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24c268f7-350f-4518-a09d-5d617a6819b1 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '676d04ef5a984edcaad63d9156568660', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:08.880857 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c268f7-350f-4518-a09d-5d617a6819b1 service placement] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:08.881419 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c268f7-350f-4518-a09d-5d617a6819b1 service placement] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:08.881920 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c268f7-350f-4518-a09d-5d617a6819b1 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:08.887000 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-24c268f7-350f-4518-a09d-5d617a6819b1 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:08.922036 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24c268f7-350f-4518-a09d-5d617a6819b1 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:08.925550 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24c268f7-350f-4518-a09d-5d617a6819b1 service placement] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:08.975359 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 864/1738] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:36:08 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 171 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:36:09.942716 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-875e3d48-11f4-421a-bcbf-78ec6a76e4b7 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:09.943235 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-875e3d48-11f4-421a-bcbf-78ec6a76e4b7 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:09.943235 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-875e3d48-11f4-421a-bcbf-78ec6a76e4b7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:10.067810 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-875e3d48-11f4-421a-bcbf-78ec6a76e4b7 None None] MFA Rules not processed for user `0268e88e0c5645a98bab5664be6e49bc`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:36:10.079074 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-875e3d48-11f4-421a-bcbf-78ec6a76e4b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:10.243805 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24c268f7-350f-4518-a09d-5d617a6819b1 service placement] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:10.245798 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-24c268f7-350f-4518-a09d-5d617a6819b1 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:10.292223 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 875/1739] 10.222.0.55 () {58 vars in 1015 bytes} [Sun Sep 25 04:36:09 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 352 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:36:10.297815 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24c268f7-350f-4518-a09d-5d617a6819b1 service placement] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:10.342386 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-875e3d48-11f4-421a-bcbf-78ec6a76e4b7 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:10.344723 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-875e3d48-11f4-421a-bcbf-78ec6a76e4b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:10.359055 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-875e3d48-11f4-421a-bcbf-78ec6a76e4b7 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:10.368335 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14518d36-120a-450b-b989-6e73f4ce2046 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:10.369289 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14518d36-120a-450b-b989-6e73f4ce2046 service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:10.369699 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14518d36-120a-450b-b989-6e73f4ce2046 service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:10.370740 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14518d36-120a-450b-b989-6e73f4ce2046 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:10.423072 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14518d36-120a-450b-b989-6e73f4ce2046 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:10.429266 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14518d36-120a-450b-b989-6e73f4ce2046 service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:10.443613 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 876/1740] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:36:10 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:36:10.518685 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd5e8c84-1bf3-4437-93c2-83f1cf59f3b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:10.519879 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd5e8c84-1bf3-4437-93c2-83f1cf59f3b4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:10.520687 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd5e8c84-1bf3-4437-93c2-83f1cf59f3b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:10.521194 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd5e8c84-1bf3-4437-93c2-83f1cf59f3b4 admin admin] PATH_INFO: `/v3/projects/413ea30d91d54a339c1514324182c553` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:10.525593 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd5e8c84-1bf3-4437-93c2-83f1cf59f3b4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=413ea30d91d54a339c1514324182c553)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:10.529890 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd5e8c84-1bf3-4437-93c2-83f1cf59f3b4 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:10.669098 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-fd5e8c84-1bf3-4437-93c2-83f1cf59f3b4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '413ea30d91d54a339c1514324182c553', 'request_id': 'req-fd5e8c84-1bf3-4437-93c2-83f1cf59f3b4'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:10.678734 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=96,keystone:DELETE=4 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:36:10.692680 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-fd5e8c84-1bf3-4437-93c2-83f1cf59f3b4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '413ea30d91d54a339c1514324182c553', 'request_id': 'req-fd5e8c84-1bf3-4437-93c2-83f1cf59f3b4'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:10.711231 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-fd5e8c84-1bf3-4437-93c2-83f1cf59f3b4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '413ea30d91d54a339c1514324182c553', 'request_id': 'req-fd5e8c84-1bf3-4437-93c2-83f1cf59f3b4'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:10.762557 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-fd5e8c84-1bf3-4437-93c2-83f1cf59f3b4 admin admin] The token cache is being invalidate because project 413ea30d91d54a339c1514324182c553 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:36:10.763104 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-fd5e8c84-1bf3-4437-93c2-83f1cf59f3b4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:10.766852 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-fd5e8c84-1bf3-4437-93c2-83f1cf59f3b4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:10.770893 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 865/1741] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:36:10 2022] DELETE /identity/v3/projects/413ea30d91d54a339c1514324182c553 => generated 0 bytes in 528 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:36:10.775124 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14518d36-120a-450b-b989-6e73f4ce2046 service neutron] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:10.798233 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14518d36-120a-450b-b989-6e73f4ce2046 service neutron] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:10.970732 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:36:10.981518 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58958c25-3042-46ad-a7c0-7dd381743241 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:10.982975 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58958c25-3042-46ad-a7c0-7dd381743241 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:10.986280 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58958c25-3042-46ad-a7c0-7dd381743241 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:10.986785 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58958c25-3042-46ad-a7c0-7dd381743241 admin admin] PATH_INFO: `/v3/users/eeb040f44f3b4312be442d539dab2583` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:10.988400 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58958c25-3042-46ad-a7c0-7dd381743241 admin admin] RBAC: Authorizing `identity:delete_user(user_id=eeb040f44f3b4312be442d539dab2583)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:10.990565 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58958c25-3042-46ad-a7c0-7dd381743241 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:11.112484 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7460878-930a-4dcd-9931-df6291ebbddc None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:11.112800 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7460878-930a-4dcd-9931-df6291ebbddc None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:11.113175 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7460878-930a-4dcd-9931-df6291ebbddc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:11.147933 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-58958c25-3042-46ad-a7c0-7dd381743241 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'eeb040f44f3b4312be442d539dab2583'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:11.175290 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-58958c25-3042-46ad-a7c0-7dd381743241 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'eeb040f44f3b4312be442d539dab2583'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:11.197861 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-c7460878-930a-4dcd-9931-df6291ebbddc None None] MFA Rules not processed for user `16281c79f73e4063b52066966c393148`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:36:11.208960 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-58958c25-3042-46ad-a7c0-7dd381743241 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'eeb040f44f3b4312be442d539dab2583'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:11.210022 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-c7460878-930a-4dcd-9931-df6291ebbddc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:11.210996 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-58958c25-3042-46ad-a7c0-7dd381743241 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'eeb040f44f3b4312be442d539dab2583'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:11.250889 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-58958c25-3042-46ad-a7c0-7dd381743241 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'eeb040f44f3b4312be442d539dab2583'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:11.265095 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 877/1742] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:36:10 2022] DELETE /identity/v3/users/eeb040f44f3b4312be442d539dab2583 => generated 0 bytes in 490 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:36:11.268102 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58958c25-3042-46ad-a7c0-7dd381743241 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:11.271220 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-58958c25-3042-46ad-a7c0-7dd381743241 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:11.351272 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 866/1743] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:36:11 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 240 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:36:11.364965 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7460878-930a-4dcd-9931-df6291ebbddc None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:11.366966 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-c7460878-930a-4dcd-9931-df6291ebbddc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:11.498320 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58958c25-3042-46ad-a7c0-7dd381743241 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:11.515370 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7460878-930a-4dcd-9931-df6291ebbddc None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:11.545663 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-084e7b67-eae0-4271-a082-2e75ce1c4618 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:11.546367 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-084e7b67-eae0-4271-a082-2e75ce1c4618 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:11.546651 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-084e7b67-eae0-4271-a082-2e75ce1c4618 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:11.546928 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-084e7b67-eae0-4271-a082-2e75ce1c4618 admin admin] PATH_INFO: `/v3/projects/dde405f4d22940388b59dfee633ae404` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:11.553748 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-084e7b67-eae0-4271-a082-2e75ce1c4618 admin admin] RBAC: Authorizing `identity:delete_project(project_id=dde405f4d22940388b59dfee633ae404)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:11.555748 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-084e7b67-eae0-4271-a082-2e75ce1c4618 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:11.733666 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-084e7b67-eae0-4271-a082-2e75ce1c4618 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'dde405f4d22940388b59dfee633ae404', 'request_id': 'req-084e7b67-eae0-4271-a082-2e75ce1c4618'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:11.739612 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=96,keystone:INSERT=1,keystone:DELETE=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:36:11.750558 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-084e7b67-eae0-4271-a082-2e75ce1c4618 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'dde405f4d22940388b59dfee633ae404', 'request_id': 'req-084e7b67-eae0-4271-a082-2e75ce1c4618'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:11.765203 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3bf8ec4-572d-4d3a-a7d8-e275b6833249 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e703b42c53974daaa608b0ccc3f018b1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'ResellerAdmin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:11.765822 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3bf8ec4-572d-4d3a-a7d8-e275b6833249 service nova] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:11.766107 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3bf8ec4-572d-4d3a-a7d8-e275b6833249 service nova] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:11.766384 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3bf8ec4-572d-4d3a-a7d8-e275b6833249 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:11.773640 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-084e7b67-eae0-4271-a082-2e75ce1c4618 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'dde405f4d22940388b59dfee633ae404', 'request_id': 'req-084e7b67-eae0-4271-a082-2e75ce1c4618'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:11.800061 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-084e7b67-eae0-4271-a082-2e75ce1c4618 admin admin] The token cache is being invalidate because project dde405f4d22940388b59dfee633ae404 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81434) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:36:11.800369 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-084e7b67-eae0-4271-a082-2e75ce1c4618 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:11.807794 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-084e7b67-eae0-4271-a082-2e75ce1c4618 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:11.822719 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 878/1744] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:36:11 2022] DELETE /identity/v3/projects/dde405f4d22940388b59dfee633ae404 => generated 0 bytes in 554 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:36:11.825649 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-084e7b67-eae0-4271-a082-2e75ce1c4618 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:11.828623 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-084e7b67-eae0-4271-a082-2e75ce1c4618 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:11.851467 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:36:11.910120 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3bf8ec4-572d-4d3a-a7d8-e275b6833249 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:11.912912 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3bf8ec4-572d-4d3a-a7d8-e275b6833249 service nova] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:11.939298 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-084e7b67-eae0-4271-a082-2e75ce1c4618 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:11.953129 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17098d46-0861-4d5b-89d0-41bd35380c43 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:11.953976 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17098d46-0861-4d5b-89d0-41bd35380c43 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:11.954404 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17098d46-0861-4d5b-89d0-41bd35380c43 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:11.954821 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17098d46-0861-4d5b-89d0-41bd35380c43 admin admin] PATH_INFO: `/v3/users/2ca34666ae954013b68c5a931898940d` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:11.960577 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 867/1745] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:36:11 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 596 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:36:11.962644 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17098d46-0861-4d5b-89d0-41bd35380c43 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2ca34666ae954013b68c5a931898940d)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:11.965244 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17098d46-0861-4d5b-89d0-41bd35380c43 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:12.063375 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-17098d46-0861-4d5b-89d0-41bd35380c43 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2ca34666ae954013b68c5a931898940d'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:12.076036 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-17098d46-0861-4d5b-89d0-41bd35380c43 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2ca34666ae954013b68c5a931898940d'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:12.097320 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-17098d46-0861-4d5b-89d0-41bd35380c43 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2ca34666ae954013b68c5a931898940d'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:12.098528 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-17098d46-0861-4d5b-89d0-41bd35380c43 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2ca34666ae954013b68c5a931898940d'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:12.108631 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-17098d46-0861-4d5b-89d0-41bd35380c43 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2ca34666ae954013b68c5a931898940d'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:12.119821 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 879/1746] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:36:11 2022] DELETE /identity/v3/users/2ca34666ae954013b68c5a931898940d => generated 0 bytes in 293 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:36:13.051122 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3bf8ec4-572d-4d3a-a7d8-e275b6833249 service nova] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:13.054067 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-b3bf8ec4-572d-4d3a-a7d8-e275b6833249 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:13.089885 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3bf8ec4-572d-4d3a-a7d8-e275b6833249 service nova] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:13.144890 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17098d46-0861-4d5b-89d0-41bd35380c43 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:13.147388 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-17098d46-0861-4d5b-89d0-41bd35380c43 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:13.290937 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14fb44ba-de6b-44fb-b1bb-c9b5d16e437f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:13.291575 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17098d46-0861-4d5b-89d0-41bd35380c43 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:13.291788 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14fb44ba-de6b-44fb-b1bb-c9b5d16e437f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:13.292110 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14fb44ba-de6b-44fb-b1bb-c9b5d16e437f admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:13.292467 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14fb44ba-de6b-44fb-b1bb-c9b5d16e437f admin admin] PATH_INFO: `/v3/projects/92acfd3b56304d59bebbda7bb72155bc` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:13.298512 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14fb44ba-de6b-44fb-b1bb-c9b5d16e437f admin admin] RBAC: Authorizing `identity:delete_project(project_id=92acfd3b56304d59bebbda7bb72155bc)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:13.300808 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14fb44ba-de6b-44fb-b1bb-c9b5d16e437f admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:13.308419 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e62a8128-dd67-4a99-a01b-d0254ad71529 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:13.309292 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e62a8128-dd67-4a99-a01b-d0254ad71529 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:13.309740 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e62a8128-dd67-4a99-a01b-d0254ad71529 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:13.310718 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e62a8128-dd67-4a99-a01b-d0254ad71529 admin admin] PATH_INFO: `/v3/projects/5ae19fc4d2ec4d4c979274be0bd3d14a` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:13.312065 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e62a8128-dd67-4a99-a01b-d0254ad71529 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5ae19fc4d2ec4d4c979274be0bd3d14a)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:13.314175 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e62a8128-dd67-4a99-a01b-d0254ad71529 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:13.392900 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:36:13.464428 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-e62a8128-dd67-4a99-a01b-d0254ad71529 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5ae19fc4d2ec4d4c979274be0bd3d14a', 'request_id': 'req-e62a8128-dd67-4a99-a01b-d0254ad71529'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:13.485715 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-e62a8128-dd67-4a99-a01b-d0254ad71529 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5ae19fc4d2ec4d4c979274be0bd3d14a', 'request_id': 'req-e62a8128-dd67-4a99-a01b-d0254ad71529'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:13.494134 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-e62a8128-dd67-4a99-a01b-d0254ad71529 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5ae19fc4d2ec4d4c979274be0bd3d14a', 'request_id': 'req-e62a8128-dd67-4a99-a01b-d0254ad71529'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:13.502771 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-14fb44ba-de6b-44fb-b1bb-c9b5d16e437f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '92acfd3b56304d59bebbda7bb72155bc', 'request_id': 'req-14fb44ba-de6b-44fb-b1bb-c9b5d16e437f'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:13.518054 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-14fb44ba-de6b-44fb-b1bb-c9b5d16e437f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '92acfd3b56304d59bebbda7bb72155bc', 'request_id': 'req-14fb44ba-de6b-44fb-b1bb-c9b5d16e437f'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:13.541444 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-e62a8128-dd67-4a99-a01b-d0254ad71529 admin admin] The token cache is being invalidate because project 5ae19fc4d2ec4d4c979274be0bd3d14a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81434) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:36:13.544112 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-14fb44ba-de6b-44fb-b1bb-c9b5d16e437f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '92acfd3b56304d59bebbda7bb72155bc', 'request_id': 'req-14fb44ba-de6b-44fb-b1bb-c9b5d16e437f'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:13.550812 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-e62a8128-dd67-4a99-a01b-d0254ad71529 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:13.553060 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-e62a8128-dd67-4a99-a01b-d0254ad71529 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:13.562512 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 880/1747] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:36:13 2022] DELETE /identity/v3/projects/5ae19fc4d2ec4d4c979274be0bd3d14a => generated 0 bytes in 414 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:36:13.565793 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bebe9328-01af-4c0a-8b22-87a5107035a6 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:13.566321 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bebe9328-01af-4c0a-8b22-87a5107035a6 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:13.566791 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bebe9328-01af-4c0a-8b22-87a5107035a6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:13.578044 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-14fb44ba-de6b-44fb-b1bb-c9b5d16e437f admin admin] The token cache is being invalidate because project 92acfd3b56304d59bebbda7bb72155bc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:36:13.578515 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-14fb44ba-de6b-44fb-b1bb-c9b5d16e437f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:13.583128 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-14fb44ba-de6b-44fb-b1bb-c9b5d16e437f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:13.586394 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 868/1748] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:36:13 2022] DELETE /identity/v3/projects/92acfd3b56304d59bebbda7bb72155bc => generated 0 bytes in 536 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:36:13.590506 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14fb44ba-de6b-44fb-b1bb-c9b5d16e437f admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:13.599052 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-14fb44ba-de6b-44fb-b1bb-c9b5d16e437f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:13.641345 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-bebe9328-01af-4c0a-8b22-87a5107035a6 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:36:13.645193 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-bebe9328-01af-4c0a-8b22-87a5107035a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:13.713614 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:36:13.903362 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14fb44ba-de6b-44fb-b1bb-c9b5d16e437f admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:13.928912 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da426a94-6ee3-448c-8b42-fa2b5a388288 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:13.934551 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da426a94-6ee3-448c-8b42-fa2b5a388288 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:13.937932 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da426a94-6ee3-448c-8b42-fa2b5a388288 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:13.939751 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da426a94-6ee3-448c-8b42-fa2b5a388288 admin admin] PATH_INFO: `/v3/users/81f27dc3f1df471ca4a49e28d7686461` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:13.942576 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da426a94-6ee3-448c-8b42-fa2b5a388288 admin admin] RBAC: Authorizing `identity:delete_user(user_id=81f27dc3f1df471ca4a49e28d7686461)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:13.950515 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da426a94-6ee3-448c-8b42-fa2b5a388288 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:13.989561 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 881/1749] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:36:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 425 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:36:14.006411 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92d0e50d-096d-494a-b1c4-840b46c9494c None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:14.006934 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92d0e50d-096d-494a-b1c4-840b46c9494c None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:14.007431 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92d0e50d-096d-494a-b1c4-840b46c9494c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:14.072295 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-92d0e50d-096d-494a-b1c4-840b46c9494c None None] MFA Rules not processed for user `c50ba6f9a164463eae9a248936e7bf80`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:36:14.082608 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-92d0e50d-096d-494a-b1c4-840b46c9494c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:14.156039 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-da426a94-6ee3-448c-8b42-fa2b5a388288 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '81f27dc3f1df471ca4a49e28d7686461'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:14.180788 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-da426a94-6ee3-448c-8b42-fa2b5a388288 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '81f27dc3f1df471ca4a49e28d7686461'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:14.242050 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-da426a94-6ee3-448c-8b42-fa2b5a388288 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '81f27dc3f1df471ca4a49e28d7686461'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:14.245676 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-da426a94-6ee3-448c-8b42-fa2b5a388288 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '81f27dc3f1df471ca4a49e28d7686461'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:14.282673 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-da426a94-6ee3-448c-8b42-fa2b5a388288 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '81f27dc3f1df471ca4a49e28d7686461'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:14.305681 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 869/1750] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:36:13 2022] DELETE /identity/v3/users/81f27dc3f1df471ca4a49e28d7686461 => generated 0 bytes in 716 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:36:14.312009 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d0602c5-ebc5-49b5-a3c1-4f4928751ac8 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:14.312395 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d0602c5-ebc5-49b5-a3c1-4f4928751ac8 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:14.312790 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d0602c5-ebc5-49b5-a3c1-4f4928751ac8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:14.344589 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 882/1751] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:36:13 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 355 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:36:14.372829 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-2d0602c5-ebc5-49b5-a3c1-4f4928751ac8 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:36:14.375844 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-2d0602c5-ebc5-49b5-a3c1-4f4928751ac8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:14.618259 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 870/1752] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:36:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 312 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:36:14.622706 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92d0e50d-096d-494a-b1c4-840b46c9494c None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:14.647323 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92d0e50d-096d-494a-b1c4-840b46c9494c None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:14.655840 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09a99f2d-9e7e-4519-9c8b-aa7ce66ab9c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:14.680909 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09a99f2d-9e7e-4519-9c8b-aa7ce66ab9c1 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:14.682388 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09a99f2d-9e7e-4519-9c8b-aa7ce66ab9c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:14.682838 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09a99f2d-9e7e-4519-9c8b-aa7ce66ab9c1 admin admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:14.688029 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09a99f2d-9e7e-4519-9c8b-aa7ce66ab9c1 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:36:14.690644 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09a99f2d-9e7e-4519-9c8b-aa7ce66ab9c1 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:14.709591 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09a99f2d-9e7e-4519-9c8b-aa7ce66ab9c1 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:14.740443 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 883/1753] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Sep 25 04:36:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 118 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:36:14.744988 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d0602c5-ebc5-49b5-a3c1-4f4928751ac8 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:14.770651 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d0602c5-ebc5-49b5-a3c1-4f4928751ac8 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:14.787205 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-287a9475-8cf2-4bd7-bc7b-31fee5b96913 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:14.788395 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-287a9475-8cf2-4bd7-bc7b-31fee5b96913 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:14.789062 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-287a9475-8cf2-4bd7-bc7b-31fee5b96913 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:14.789673 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-287a9475-8cf2-4bd7-bc7b-31fee5b96913 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:14.790570 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-287a9475-8cf2-4bd7-bc7b-31fee5b96913 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:14.795733 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-287a9475-8cf2-4bd7-bc7b-31fee5b96913 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:14.880227 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 871/1754] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:36:14 2022] POST /identity/v3/projects => generated 355 bytes in 136 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:36:14.884569 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09a99f2d-9e7e-4519-9c8b-aa7ce66ab9c1 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:14.903611 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09a99f2d-9e7e-4519-9c8b-aa7ce66ab9c1 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:15.021458 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:36:15.058985 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e28148e2-57dd-4400-9ac0-97fe8515dce8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:15.060131 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e28148e2-57dd-4400-9ac0-97fe8515dce8 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:15.062401 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e28148e2-57dd-4400-9ac0-97fe8515dce8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:15.063114 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e28148e2-57dd-4400-9ac0-97fe8515dce8 admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:15.064211 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e28148e2-57dd-4400-9ac0-97fe8515dce8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:15.066906 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e28148e2-57dd-4400-9ac0-97fe8515dce8 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:15.106279 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 884/1755] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:36:14 2022] POST /identity/v3/users => generated 278 bytes in 220 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 04:36:15.107754 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-287a9475-8cf2-4bd7-bc7b-31fee5b96913 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:15.156193 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-287a9475-8cf2-4bd7-bc7b-31fee5b96913 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:15.169049 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f44c37fe-9675-406b-a955-01c563114635 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:15.170903 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f44c37fe-9675-406b-a955-01c563114635 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:15.171188 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f44c37fe-9675-406b-a955-01c563114635 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:15.171463 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f44c37fe-9675-406b-a955-01c563114635 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:15.172895 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f44c37fe-9675-406b-a955-01c563114635 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:15.177111 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f44c37fe-9675-406b-a955-01c563114635 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:15.199526 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 872/1756] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:36:15 2022] GET /identity/v3/roles => generated 1420 bytes in 90 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:36:15.200800 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e28148e2-57dd-4400-9ac0-97fe8515dce8 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:15.215392 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e28148e2-57dd-4400-9ac0-97fe8515dce8 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:15.226462 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6679c45-315a-42a8-89be-d1c71fff998b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:15.227049 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6679c45-315a-42a8-89be-d1c71fff998b admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:15.227332 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6679c45-315a-42a8-89be-d1c71fff998b admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:15.227684 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6679c45-315a-42a8-89be-d1c71fff998b admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:15.228337 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6679c45-315a-42a8-89be-d1c71fff998b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:15.229992 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6679c45-315a-42a8-89be-d1c71fff998b admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:15.270383 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 885/1757] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:36:15 2022] GET /identity/v3/roles => generated 1420 bytes in 70 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:36:15.273374 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f44c37fe-9675-406b-a955-01c563114635 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:15.287844 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f44c37fe-9675-406b-a955-01c563114635 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:15.300726 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2608ad31-3852-4fae-b96a-de3391672d6e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:15.301419 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2608ad31-3852-4fae-b96a-de3391672d6e admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:15.301696 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2608ad31-3852-4fae-b96a-de3391672d6e admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:15.301971 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2608ad31-3852-4fae-b96a-de3391672d6e admin admin] PATH_INFO: `/v3/projects/26d0487282904512bffc6b4c356bcdd8/users/60ff17baa7ca44c5b444af1ad8565e33/roles/8f6cc574056c4278bc9778f14dc4b411` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:15.338721 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2608ad31-3852-4fae-b96a-de3391672d6e admin admin] RBAC: Authorizing `identity:create_grant(project_id=26d0487282904512bffc6b4c356bcdd8, user_id=60ff17baa7ca44c5b444af1ad8565e33, role_id=8f6cc574056c4278bc9778f14dc4b411)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:15.341649 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2608ad31-3852-4fae-b96a-de3391672d6e admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:15.344916 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:36:15.381750 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 873/1758] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:36:15 2022] PUT /identity/v3/projects/26d0487282904512bffc6b4c356bcdd8/users/60ff17baa7ca44c5b444af1ad8565e33/roles/8f6cc574056c4278bc9778f14dc4b411 => generated 0 bytes in 100 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 04:36:15.392514 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b79e1896-79e2-4d54-9321-8fc4fae05a70 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:15.393163 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b79e1896-79e2-4d54-9321-8fc4fae05a70 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:15.394385 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b79e1896-79e2-4d54-9321-8fc4fae05a70 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:15.566315 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-b79e1896-79e2-4d54-9321-8fc4fae05a70 None None] MFA Rules not processed for user `60ff17baa7ca44c5b444af1ad8565e33`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:36:15.570569 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-b79e1896-79e2-4d54-9321-8fc4fae05a70 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:15.609653 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2608ad31-3852-4fae-b96a-de3391672d6e admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:15.612105 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-2608ad31-3852-4fae-b96a-de3391672d6e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:15.814149 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2608ad31-3852-4fae-b96a-de3391672d6e admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:15.835642 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-121beddf-013e-4fcf-a1ac-33a2bf88a8b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:15.836311 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-121beddf-013e-4fcf-a1ac-33a2bf88a8b0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:15.836626 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-121beddf-013e-4fcf-a1ac-33a2bf88a8b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:15.836936 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-121beddf-013e-4fcf-a1ac-33a2bf88a8b0 admin admin] PATH_INFO: `/v3/projects/706b8c4a2f4446aea67f6be7a17d56dd` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:15.838073 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-121beddf-013e-4fcf-a1ac-33a2bf88a8b0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=706b8c4a2f4446aea67f6be7a17d56dd)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:15.840002 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-121beddf-013e-4fcf-a1ac-33a2bf88a8b0 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:15.849462 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 886/1759] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:36:15 2022] POST /identity/v3/auth/tokens => generated 3232 bytes in 467 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:36:15.850975 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad95a494-b91f-4591-8bf8-f62a34f5dcfc None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:15.854749 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad95a494-b91f-4591-8bf8-f62a34f5dcfc None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:15.854749 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad95a494-b91f-4591-8bf8-f62a34f5dcfc None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:15.858746 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 887/1760] 10.222.0.25 () {54 vars in 921 bytes} [Sun Sep 25 04:36:15 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:36:15.863003 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad95a494-b91f-4591-8bf8-f62a34f5dcfc None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:15.937770 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad95a494-b91f-4591-8bf8-f62a34f5dcfc None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:15.961669 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4eaae88-5986-4ae1-bb73-6050f928a5cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:15.962320 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4eaae88-5986-4ae1-bb73-6050f928a5cf admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:15.962608 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4eaae88-5986-4ae1-bb73-6050f928a5cf admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:15.962882 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4eaae88-5986-4ae1-bb73-6050f928a5cf admin admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:15.983290 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4eaae88-5986-4ae1-bb73-6050f928a5cf admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:15.988896 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4eaae88-5986-4ae1-bb73-6050f928a5cf admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:15.994544 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-121beddf-013e-4fcf-a1ac-33a2bf88a8b0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '706b8c4a2f4446aea67f6be7a17d56dd', 'request_id': 'req-121beddf-013e-4fcf-a1ac-33a2bf88a8b0'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:16.011560 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-121beddf-013e-4fcf-a1ac-33a2bf88a8b0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '706b8c4a2f4446aea67f6be7a17d56dd', 'request_id': 'req-121beddf-013e-4fcf-a1ac-33a2bf88a8b0'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:16.041781 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-121beddf-013e-4fcf-a1ac-33a2bf88a8b0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '706b8c4a2f4446aea67f6be7a17d56dd', 'request_id': 'req-121beddf-013e-4fcf-a1ac-33a2bf88a8b0'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:16.064843 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-121beddf-013e-4fcf-a1ac-33a2bf88a8b0 admin admin] The token cache is being invalidate because project 706b8c4a2f4446aea67f6be7a17d56dd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:36:16.065771 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-121beddf-013e-4fcf-a1ac-33a2bf88a8b0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:16.074549 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-121beddf-013e-4fcf-a1ac-33a2bf88a8b0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:16.082231 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 874/1761] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:36:15 2022] DELETE /identity/v3/projects/706b8c4a2f4446aea67f6be7a17d56dd => generated 0 bytes in 469 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:36:16.084125 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a51a92-6396-4c22-8a1a-983aa5f4c306 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:16.084477 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a51a92-6396-4c22-8a1a-983aa5f4c306 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:16.084849 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a51a92-6396-4c22-8a1a-983aa5f4c306 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:16.136856 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-85a51a92-6396-4c22-8a1a-983aa5f4c306 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:36:16.136856 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-85a51a92-6396-4c22-8a1a-983aa5f4c306 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:16.155844 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 888/1762] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:36:15 2022] POST /identity/v3/projects => generated 357 bytes in 293 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:36:16.155844 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-633fa09f-c242-427b-a105-369f8d3da5a1 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:16.155844 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-633fa09f-c242-427b-a105-369f8d3da5a1 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:16.155844 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-633fa09f-c242-427b-a105-369f8d3da5a1 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:16.175213 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 889/1763] 10.222.0.25 () {54 vars in 921 bytes} [Sun Sep 25 04:36:16 2022] GET /identity => generated 270 bytes in 22 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:36:16.176293 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-633fa09f-c242-427b-a105-369f8d3da5a1 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:16.185956 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-633fa09f-c242-427b-a105-369f8d3da5a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:16.340500 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 875/1764] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:36:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 251 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 04:36:16.340500 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec692c08-0eb9-4795-baaf-de0a207b4654 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:16.340500 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec692c08-0eb9-4795-baaf-de0a207b4654 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:16.340500 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec692c08-0eb9-4795-baaf-de0a207b4654 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:16.426027 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-ec692c08-0eb9-4795-baaf-de0a207b4654 None None] MFA Rules not processed for user `e703b42c53974daaa608b0ccc3f018b1`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:36:16.427301 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-ec692c08-0eb9-4795-baaf-de0a207b4654 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:16.501391 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-633fa09f-c242-427b-a105-369f8d3da5a1 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:16.534626 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6abb2b5a-ffe6-4dcb-af32-fad41839b280 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:16.535065 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:INSERT=2,keystone:SELECT=91,keystone:DELETE=7 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:36:16.535315 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6abb2b5a-ffe6-4dcb-af32-fad41839b280 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:16.535576 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6abb2b5a-ffe6-4dcb-af32-fad41839b280 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:16.535859 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6abb2b5a-ffe6-4dcb-af32-fad41839b280 admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:16.536439 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6abb2b5a-ffe6-4dcb-af32-fad41839b280 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:16.538464 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6abb2b5a-ffe6-4dcb-af32-fad41839b280 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:16.631374 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 890/1765] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:36:16 2022] POST /identity/v3/users => generated 285 bytes in 453 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:36:16.631374 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a15a481b-fc78-4e25-a317-0642d1e66b0d None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:16.632516 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a15a481b-fc78-4e25-a317-0642d1e66b0d None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:16.632837 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a15a481b-fc78-4e25-a317-0642d1e66b0d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:16.717652 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-a15a481b-fc78-4e25-a317-0642d1e66b0d None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:36:16.723184 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-a15a481b-fc78-4e25-a317-0642d1e66b0d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:16.727570 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 876/1766] 10.222.0.25 () {58 vars in 1018 bytes} [Sun Sep 25 04:36:16 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 393 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:36:16.730334 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ecdce54-086c-468e-ac1f-d58fdf077094 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:16.730645 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ecdce54-086c-468e-ac1f-d58fdf077094 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:16.734837 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ecdce54-086c-468e-ac1f-d58fdf077094 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:16.758700 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 891/1767] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:36:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:36:16.761358 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a15a481b-fc78-4e25-a317-0642d1e66b0d None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:16.791925 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a15a481b-fc78-4e25-a317-0642d1e66b0d None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:16.808020 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d4a3a2e-387b-47f5-8d30-3fe2e907bdcb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:16.808756 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d4a3a2e-387b-47f5-8d30-3fe2e907bdcb admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:16.809254 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d4a3a2e-387b-47f5-8d30-3fe2e907bdcb admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:16.809365 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d4a3a2e-387b-47f5-8d30-3fe2e907bdcb admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:16.810265 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d4a3a2e-387b-47f5-8d30-3fe2e907bdcb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:16.812424 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d4a3a2e-387b-47f5-8d30-3fe2e907bdcb admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:16.821334 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-5ecdce54-086c-468e-ac1f-d58fdf077094 None None] MFA Rules not processed for user `e703b42c53974daaa608b0ccc3f018b1`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:36:16.823561 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-5ecdce54-086c-468e-ac1f-d58fdf077094 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:16.843096 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 892/1768] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:36:16 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:36:16.845872 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d4a3a2e-387b-47f5-8d30-3fe2e907bdcb admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:16.870585 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d4a3a2e-387b-47f5-8d30-3fe2e907bdcb admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:16.884784 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cb45ca4-e175-49e6-9cbd-842a0c2a9f40 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:16.885416 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb45ca4-e175-49e6-9cbd-842a0c2a9f40 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:16.885699 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb45ca4-e175-49e6-9cbd-842a0c2a9f40 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:16.885975 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb45ca4-e175-49e6-9cbd-842a0c2a9f40 admin admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:16.886626 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cb45ca4-e175-49e6-9cbd-842a0c2a9f40 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:36:16.886929 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cb45ca4-e175-49e6-9cbd-842a0c2a9f40 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:16.888939 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 877/1769] 10.222.0.25 () {58 vars in 1018 bytes} [Sun Sep 25 04:36:16 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:36:16.891040 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cb45ca4-e175-49e6-9cbd-842a0c2a9f40 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:16.891307 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ecdce54-086c-468e-ac1f-d58fdf077094 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:16.893061 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:36:16.917136 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ecdce54-086c-468e-ac1f-d58fdf077094 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:16.924036 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 893/1770] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Sep 25 04:36:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:36:16.927956 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cb45ca4-e175-49e6-9cbd-842a0c2a9f40 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:16.933026 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23c003ef-8875-43e1-ae79-5055283108d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:16.937422 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23c003ef-8875-43e1-ae79-5055283108d0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:16.937732 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23c003ef-8875-43e1-ae79-5055283108d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:16.959906 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cb45ca4-e175-49e6-9cbd-842a0c2a9f40 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:16.964747 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23c003ef-8875-43e1-ae79-5055283108d0 admin admin] PATH_INFO: `/v3/projects/26d4a30838dc41fb923f18b7aa73a3d9/users/95f823c6c5f844ddbfcd45b4afbb203e/roles/c51fe7c5ad804040879acfb54269455f` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:16.972592 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bb1eb86-7b48-4322-b046-d99b1986cf73 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:16.973475 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bb1eb86-7b48-4322-b046-d99b1986cf73 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:16.973843 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bb1eb86-7b48-4322-b046-d99b1986cf73 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:16.974698 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bb1eb86-7b48-4322-b046-d99b1986cf73 admin admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:16.975358 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bb1eb86-7b48-4322-b046-d99b1986cf73 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:16.977304 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bb1eb86-7b48-4322-b046-d99b1986cf73 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:17.017269 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23c003ef-8875-43e1-ae79-5055283108d0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=26d4a30838dc41fb923f18b7aa73a3d9, user_id=95f823c6c5f844ddbfcd45b4afbb203e, role_id=c51fe7c5ad804040879acfb54269455f)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:17.045046 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23c003ef-8875-43e1-ae79-5055283108d0 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:17.047527 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 894/1771] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:36:16 2022] POST /identity/v3/projects => generated 401 bytes in 120 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:36:17.051391 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bb1eb86-7b48-4322-b046-d99b1986cf73 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:17.119389 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 878/1772] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:36:16 2022] PUT /identity/v3/projects/26d4a30838dc41fb923f18b7aa73a3d9/users/95f823c6c5f844ddbfcd45b4afbb203e/roles/c51fe7c5ad804040879acfb54269455f => generated 0 bytes in 230 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:36:17.124781 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23c003ef-8875-43e1-ae79-5055283108d0 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:17.230172 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bb1eb86-7b48-4322-b046-d99b1986cf73 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:17.359085 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bfa81db-f5af-45fa-82df-6905558309aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:17.360382 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bfa81db-f5af-45fa-82df-6905558309aa admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:17.360831 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bfa81db-f5af-45fa-82df-6905558309aa admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:17.361820 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bfa81db-f5af-45fa-82df-6905558309aa admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:17.362649 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bfa81db-f5af-45fa-82df-6905558309aa admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:17.365234 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23c003ef-8875-43e1-ae79-5055283108d0 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:17.367485 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bfa81db-f5af-45fa-82df-6905558309aa admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:17.396878 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cac459b-2be0-4938-a644-8f1c1e5dc90b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:17.404377 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cac459b-2be0-4938-a644-8f1c1e5dc90b admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:17.410940 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cac459b-2be0-4938-a644-8f1c1e5dc90b admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:17.411529 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 895/1773] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:36:17 2022] POST /identity/v3/users => generated 301 bytes in 360 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:36:17.415148 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cac459b-2be0-4938-a644-8f1c1e5dc90b admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:17.415635 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bfa81db-f5af-45fa-82df-6905558309aa admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:17.416119 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cac459b-2be0-4938-a644-8f1c1e5dc90b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:17.418049 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cac459b-2be0-4938-a644-8f1c1e5dc90b admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:17.432323 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 879/1774] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:36:17 2022] GET /identity/v3/roles => generated 1420 bytes in 309 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:36:17.438360 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cac459b-2be0-4938-a644-8f1c1e5dc90b admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:17.444864 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bfa81db-f5af-45fa-82df-6905558309aa admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:17.461406 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cac459b-2be0-4938-a644-8f1c1e5dc90b admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:17.475894 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f287683-6393-46c7-b629-b48db67daab4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:17.476916 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f287683-6393-46c7-b629-b48db67daab4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:17.477381 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f287683-6393-46c7-b629-b48db67daab4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:17.477887 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f287683-6393-46c7-b629-b48db67daab4 admin admin] PATH_INFO: `/v3/projects/26d4a30838dc41fb923f18b7aa73a3d9/users/95f823c6c5f844ddbfcd45b4afbb203e/roles/c51fe7c5ad804040879acfb54269455f` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:17.486961 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f287683-6393-46c7-b629-b48db67daab4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=26d4a30838dc41fb923f18b7aa73a3d9, user_id=95f823c6c5f844ddbfcd45b4afbb203e, role_id=c51fe7c5ad804040879acfb54269455f)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:17.488105 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40575817-c860-4d6d-b983-87769b39e449 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:17.489540 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40575817-c860-4d6d-b983-87769b39e449 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:17.489540 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40575817-c860-4d6d-b983-87769b39e449 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:17.489540 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40575817-c860-4d6d-b983-87769b39e449 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:17.490686 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40575817-c860-4d6d-b983-87769b39e449 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:17.493366 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40575817-c860-4d6d-b983-87769b39e449 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:17.494425 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f287683-6393-46c7-b629-b48db67daab4 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:17.519498 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 880/1775] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:36:17 2022] PUT /identity/v3/projects/26d4a30838dc41fb923f18b7aa73a3d9/users/95f823c6c5f844ddbfcd45b4afbb203e/roles/c51fe7c5ad804040879acfb54269455f => generated 0 bytes in 82 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:36:17.530820 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-736c3020-727a-4b4f-b82b-3a48399518af None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:17.531496 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-736c3020-727a-4b4f-b82b-3a48399518af None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:17.531953 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-736c3020-727a-4b4f-b82b-3a48399518af None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:17.533079 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 896/1776] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:36:17 2022] GET /identity/v3/roles => generated 1420 bytes in 118 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:36:17.536917 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40575817-c860-4d6d-b983-87769b39e449 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:17.579572 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40575817-c860-4d6d-b983-87769b39e449 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:17.600630 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-736c3020-727a-4b4f-b82b-3a48399518af None None] MFA Rules not processed for user `95f823c6c5f844ddbfcd45b4afbb203e`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:36:17.602207 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-736c3020-727a-4b4f-b82b-3a48399518af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:17.746551 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:36:17.755430 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34dd48ff-8f65-4013-be4a-6e74f4bf9da5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:17.756481 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34dd48ff-8f65-4013-be4a-6e74f4bf9da5 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:17.757727 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34dd48ff-8f65-4013-be4a-6e74f4bf9da5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:17.758942 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34dd48ff-8f65-4013-be4a-6e74f4bf9da5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:17.759944 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34dd48ff-8f65-4013-be4a-6e74f4bf9da5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:17.762357 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:36:17.763637 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34dd48ff-8f65-4013-be4a-6e74f4bf9da5 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:17.781045 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 897/1777] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:36:17 2022] GET /identity/v3/roles => generated 1420 bytes in 243 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:36:17.785232 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34dd48ff-8f65-4013-be4a-6e74f4bf9da5 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:17.790452 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 881/1778] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:36:17 2022] POST /identity/v3/auth/tokens => generated 3301 bytes in 261 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 04:36:17.798559 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-736c3020-727a-4b4f-b82b-3a48399518af None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:17.800909 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-736c3020-727a-4b4f-b82b-3a48399518af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:17.806752 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34dd48ff-8f65-4013-be4a-6e74f4bf9da5 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:17.826569 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3e5e73e-35a4-4149-805c-d6e683831ee8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:17.827285 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3e5e73e-35a4-4149-805c-d6e683831ee8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:17.827473 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3e5e73e-35a4-4149-805c-d6e683831ee8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:17.827752 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3e5e73e-35a4-4149-805c-d6e683831ee8 admin admin] PATH_INFO: `/v3/projects/e090fa2d0a824c9c9e6081cc7928ebb8/users/c9383b84dae04d02bfac17dfa7fc8a25/roles/8f6cc574056c4278bc9778f14dc4b411` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:17.868990 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3e5e73e-35a4-4149-805c-d6e683831ee8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e090fa2d0a824c9c9e6081cc7928ebb8, user_id=c9383b84dae04d02bfac17dfa7fc8a25, role_id=8f6cc574056c4278bc9778f14dc4b411)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:17.876131 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3e5e73e-35a4-4149-805c-d6e683831ee8 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:17.893967 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 898/1779] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:36:17 2022] PUT /identity/v3/projects/e090fa2d0a824c9c9e6081cc7928ebb8/users/c9383b84dae04d02bfac17dfa7fc8a25/roles/8f6cc574056c4278bc9778f14dc4b411 => generated 0 bytes in 111 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:36:17.906515 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a98898-9f17-43e3-9c74-986307818647 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:17.906888 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a98898-9f17-43e3-9c74-986307818647 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:17.907240 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a98898-9f17-43e3-9c74-986307818647 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:17.973337 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-736c3020-727a-4b4f-b82b-3a48399518af None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:17.993492 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-35a98898-9f17-43e3-9c74-986307818647 None None] MFA Rules not processed for user `c9383b84dae04d02bfac17dfa7fc8a25`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:36:17.999525 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-35a98898-9f17-43e3-9c74-986307818647 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:18.071842 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92b9f5cc-f485-4332-80e9-b044fe18745c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ffbc136c77b84e8c87128f7247f4ec03', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:18.072615 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b9f5cc-f485-4332-80e9-b044fe18745c service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:18.073017 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b9f5cc-f485-4332-80e9-b044fe18745c service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:18.073264 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b9f5cc-f485-4332-80e9-b044fe18745c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:18.130579 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 899/1780] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:36:17 2022] POST /identity/v3/auth/tokens => generated 3278 bytes in 222 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 04:36:18.130579 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1ad6ba7-0e0d-4a60-bcbe-84dd5023cc46 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:18.130579 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1ad6ba7-0e0d-4a60-bcbe-84dd5023cc46 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:18.130579 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1ad6ba7-0e0d-4a60-bcbe-84dd5023cc46 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:18.160817 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-c1ad6ba7-0e0d-4a60-bcbe-84dd5023cc46 None None] MFA Rules not processed for user `16281c79f73e4063b52066966c393148`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:36:18.162169 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-c1ad6ba7-0e0d-4a60-bcbe-84dd5023cc46 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:18.182687 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92b9f5cc-f485-4332-80e9-b044fe18745c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:18.184356 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92b9f5cc-f485-4332-80e9-b044fe18745c service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:18.199483 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 882/1781] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:36:17 2022] GET /identity/v3/auth/tokens => generated 3301 bytes in 401 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 25 04:36:18.201288 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92b9f5cc-f485-4332-80e9-b044fe18745c service cinder] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:18.348661 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 900/1782] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:36:18 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 220 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:36:18.349806 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1ad6ba7-0e0d-4a60-bcbe-84dd5023cc46 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:18.372293 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1ad6ba7-0e0d-4a60-bcbe-84dd5023cc46 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:18.385614 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b59e20b-bc64-41a7-862f-5e38eb9adb02 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ffbc136c77b84e8c87128f7247f4ec03', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:18.386177 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b59e20b-bc64-41a7-862f-5e38eb9adb02 service cinder] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:18.386463 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b59e20b-bc64-41a7-862f-5e38eb9adb02 service cinder] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:18.386741 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b59e20b-bc64-41a7-862f-5e38eb9adb02 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:18.390220 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-7b59e20b-bc64-41a7-862f-5e38eb9adb02 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:18.391284 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92b9f5cc-f485-4332-80e9-b044fe18745c service cinder] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:18.399696 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e44d86f2-a8c8-4ece-a0fa-658819af06c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:18.401168 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e44d86f2-a8c8-4ece-a0fa-658819af06c7 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:18.401449 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e44d86f2-a8c8-4ece-a0fa-658819af06c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:18.401835 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e44d86f2-a8c8-4ece-a0fa-658819af06c7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:18.402492 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e44d86f2-a8c8-4ece-a0fa-658819af06c7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:18.405689 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e44d86f2-a8c8-4ece-a0fa-658819af06c7 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:18.413701 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:36:18.453674 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 883/1783] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:36:18 2022] POST /identity/v3/projects => generated 399 bytes in 253 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:36:18.458294 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e44d86f2-a8c8-4ece-a0fa-658819af06c7 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:18.513215 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b59e20b-bc64-41a7-862f-5e38eb9adb02 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:18.516055 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b59e20b-bc64-41a7-862f-5e38eb9adb02 service cinder] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:18.557317 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 901/1784] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:36:18 2022] GET /identity/v3/auth/tokens => generated 3232 bytes in 207 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:36:18.607107 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b59e20b-bc64-41a7-862f-5e38eb9adb02 service cinder] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:18.610384 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-7b59e20b-bc64-41a7-862f-5e38eb9adb02 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:18.617798 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:36:18.620855 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e44d86f2-a8c8-4ece-a0fa-658819af06c7 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:18.638516 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dfd1e42-7fd8-4a9c-85e4-2d88ffda9007 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:18.639993 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dfd1e42-7fd8-4a9c-85e4-2d88ffda9007 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:18.641225 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dfd1e42-7fd8-4a9c-85e4-2d88ffda9007 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:18.641765 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dfd1e42-7fd8-4a9c-85e4-2d88ffda9007 admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:18.643281 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dfd1e42-7fd8-4a9c-85e4-2d88ffda9007 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:18.651339 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dfd1e42-7fd8-4a9c-85e4-2d88ffda9007 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:18.704795 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 884/1785] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:36:18 2022] POST /identity/v3/users => generated 306 bytes in 247 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:36:18.706632 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c1f9f1f-d6bf-44c9-b56a-4f4c43ed3e7c None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:18.707347 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c1f9f1f-d6bf-44c9-b56a-4f4c43ed3e7c None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:18.708038 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c1f9f1f-d6bf-44c9-b56a-4f4c43ed3e7c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:18.739618 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-6c1f9f1f-d6bf-44c9-b56a-4f4c43ed3e7c None None] MFA Rules not processed for user `e7354aa144df4ad69cd7d12f11b1d8a5`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:36:18.741802 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-6c1f9f1f-d6bf-44c9-b56a-4f4c43ed3e7c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:18.865167 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b59e20b-bc64-41a7-862f-5e38eb9adb02 service cinder] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:18.871361 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 885/1786] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:36:18 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:36:18.871361 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c1f9f1f-d6bf-44c9-b56a-4f4c43ed3e7c None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:18.896991 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c1f9f1f-d6bf-44c9-b56a-4f4c43ed3e7c None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:18.903092 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ed8635b-dbef-4dda-a399-ff75acc159ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:18.903790 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed8635b-dbef-4dda-a399-ff75acc159ea admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:18.904254 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed8635b-dbef-4dda-a399-ff75acc159ea admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:18.904533 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed8635b-dbef-4dda-a399-ff75acc159ea admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:18.905282 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ed8635b-dbef-4dda-a399-ff75acc159ea admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:18.906865 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ed8635b-dbef-4dda-a399-ff75acc159ea admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:18.923558 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 886/1787] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:36:18 2022] GET /identity/v3/roles => generated 1420 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:36:18.924145 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ed8635b-dbef-4dda-a399-ff75acc159ea admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:18.926447 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-3ed8635b-dbef-4dda-a399-ff75acc159ea admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:19.051016 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a294f9f4-5c2c-4d5f-91a2-d9d78065416c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e703b42c53974daaa608b0ccc3f018b1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'ResellerAdmin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:19.051728 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a294f9f4-5c2c-4d5f-91a2-d9d78065416c service nova] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:19.052078 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a294f9f4-5c2c-4d5f-91a2-d9d78065416c service nova] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:19.052422 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a294f9f4-5c2c-4d5f-91a2-d9d78065416c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:19.066027 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ed8635b-dbef-4dda-a399-ff75acc159ea admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:19.071274 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a294f9f4-5c2c-4d5f-91a2-d9d78065416c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:19.073518 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a294f9f4-5c2c-4d5f-91a2-d9d78065416c service nova] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:19.085055 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21a89c46-f924-4b0f-ae90-9a0b12740953 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e703b42c53974daaa608b0ccc3f018b1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'ResellerAdmin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:19.086108 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21a89c46-f924-4b0f-ae90-9a0b12740953 service nova] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:19.086531 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21a89c46-f924-4b0f-ae90-9a0b12740953 service nova] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:19.086794 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 902/1788] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:36:18 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 481 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:36:19.087155 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21a89c46-f924-4b0f-ae90-9a0b12740953 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:19.087823 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a294f9f4-5c2c-4d5f-91a2-d9d78065416c service nova] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:19.096687 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:36:19.103316 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a294f9f4-5c2c-4d5f-91a2-d9d78065416c service nova] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:19.105135 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21a89c46-f924-4b0f-ae90-9a0b12740953 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:19.109390 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21a89c46-f924-4b0f-ae90-9a0b12740953 service nova] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:19.111213 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e53df633-45be-4bc1-b483-375a09857257 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:19.112335 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e53df633-45be-4bc1-b483-375a09857257 admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:19.112853 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e53df633-45be-4bc1-b483-375a09857257 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:19.113203 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e53df633-45be-4bc1-b483-375a09857257 admin admin] PATH_INFO: `/v3/projects/b48acfe4b8104d34859cfafc7c538f02/users/2d429f764c6d4a2e830e9c278cbf9c79/roles/c51fe7c5ad804040879acfb54269455f` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:19.132264 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 887/1789] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:36:18 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 209 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:36:19.163498 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e53df633-45be-4bc1-b483-375a09857257 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b48acfe4b8104d34859cfafc7c538f02, user_id=2d429f764c6d4a2e830e9c278cbf9c79, role_id=c51fe7c5ad804040879acfb54269455f)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:19.166736 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e53df633-45be-4bc1-b483-375a09857257 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:19.199759 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 903/1790] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:36:19 2022] PUT /identity/v3/projects/b48acfe4b8104d34859cfafc7c538f02/users/2d429f764c6d4a2e830e9c278cbf9c79/roles/c51fe7c5ad804040879acfb54269455f => generated 0 bytes in 112 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:36:19.204900 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21a89c46-f924-4b0f-ae90-9a0b12740953 service nova] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:19.233018 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21a89c46-f924-4b0f-ae90-9a0b12740953 service nova] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:19.375053 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8585f8a-ec54-4c5e-9a03-bfff0ae88deb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:19.387211 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8585f8a-ec54-4c5e-9a03-bfff0ae88deb admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:19.389713 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8585f8a-ec54-4c5e-9a03-bfff0ae88deb admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:19.390110 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8585f8a-ec54-4c5e-9a03-bfff0ae88deb admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:19.391240 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8585f8a-ec54-4c5e-9a03-bfff0ae88deb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:19.396348 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8585f8a-ec54-4c5e-9a03-bfff0ae88deb admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:19.427415 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e53df633-45be-4bc1-b483-375a09857257 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:19.445476 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e53df633-45be-4bc1-b483-375a09857257 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:19.446178 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 888/1791] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:36:19 2022] GET /identity/v3/roles => generated 1420 bytes in 239 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:36:19.455112 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-974f85d1-8580-4b59-91fa-bddfcaaa6872 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:19.455811 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-974f85d1-8580-4b59-91fa-bddfcaaa6872 admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:19.456095 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-974f85d1-8580-4b59-91fa-bddfcaaa6872 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:19.456370 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-974f85d1-8580-4b59-91fa-bddfcaaa6872 admin admin] PATH_INFO: `/v3/projects/b48acfe4b8104d34859cfafc7c538f02/users/2d429f764c6d4a2e830e9c278cbf9c79/roles/c51fe7c5ad804040879acfb54269455f` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:19.460856 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-974f85d1-8580-4b59-91fa-bddfcaaa6872 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b48acfe4b8104d34859cfafc7c538f02, user_id=2d429f764c6d4a2e830e9c278cbf9c79, role_id=c51fe7c5ad804040879acfb54269455f)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:19.466993 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-974f85d1-8580-4b59-91fa-bddfcaaa6872 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:19.511395 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 904/1792] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:36:19 2022] PUT /identity/v3/projects/b48acfe4b8104d34859cfafc7c538f02/users/2d429f764c6d4a2e830e9c278cbf9c79/roles/c51fe7c5ad804040879acfb54269455f => generated 0 bytes in 76 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:36:19.514574 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25a13a4e-957f-44b9-bacc-e21038c4d50c None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:19.514854 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25a13a4e-957f-44b9-bacc-e21038c4d50c None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:19.515141 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25a13a4e-957f-44b9-bacc-e21038c4d50c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:19.563102 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-25a13a4e-957f-44b9-bacc-e21038c4d50c None None] MFA Rules not processed for user `2d429f764c6d4a2e830e9c278cbf9c79`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:36:19.564577 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-25a13a4e-957f-44b9-bacc-e21038c4d50c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:19.736511 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 889/1793] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:36:19 2022] POST /identity/v3/auth/tokens => generated 3343 bytes in 224 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:36:19.747765 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-974f85d1-8580-4b59-91fa-bddfcaaa6872 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:19.750334 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-974f85d1-8580-4b59-91fa-bddfcaaa6872 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:19.914096 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:36:19.924844 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-974f85d1-8580-4b59-91fa-bddfcaaa6872 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:19.933333 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e563da1-19db-45d6-8808-7f9a7d589cca service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ffbc136c77b84e8c87128f7247f4ec03', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:19.933970 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e563da1-19db-45d6-8808-7f9a7d589cca service cinder] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:19.934250 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e563da1-19db-45d6-8808-7f9a7d589cca service cinder] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:19.935753 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e563da1-19db-45d6-8808-7f9a7d589cca service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:19.949796 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e563da1-19db-45d6-8808-7f9a7d589cca service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:19.954424 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e563da1-19db-45d6-8808-7f9a7d589cca service cinder] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:19.971593 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 905/1794] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:36:19 2022] GET /identity/v3/auth/tokens => generated 3343 bytes in 224 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:36:20.146183 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25a13a4e-957f-44b9-bacc-e21038c4d50c None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:20.190562 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25a13a4e-957f-44b9-bacc-e21038c4d50c None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:20.201365 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2913001-ad10-4186-871b-4a927d599f96 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ffbc136c77b84e8c87128f7247f4ec03', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:20.204354 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2913001-ad10-4186-871b-4a927d599f96 service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:20.205740 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2913001-ad10-4186-871b-4a927d599f96 service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:20.205932 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2913001-ad10-4186-871b-4a927d599f96 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:20.363267 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2913001-ad10-4186-871b-4a927d599f96 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:20.366911 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2913001-ad10-4186-871b-4a927d599f96 service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:20.411429 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 890/1795] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:36:20 2022] GET /identity/v3/auth/tokens => generated 3278 bytes in 265 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:36:29.204519 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e563da1-19db-45d6-8808-7f9a7d589cca service cinder] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:29.268873 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e563da1-19db-45d6-8808-7f9a7d589cca service cinder] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:29.273569 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5293db65-316a-480e-8029-483fb8f0b23d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:29.274214 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5293db65-316a-480e-8029-483fb8f0b23d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:29.274491 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5293db65-316a-480e-8029-483fb8f0b23d admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:29.274765 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5293db65-316a-480e-8029-483fb8f0b23d admin admin] PATH_INFO: `/v3/users/c9383b84dae04d02bfac17dfa7fc8a25` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:29.275919 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5293db65-316a-480e-8029-483fb8f0b23d admin admin] RBAC: Authorizing `identity:delete_user(user_id=c9383b84dae04d02bfac17dfa7fc8a25)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:29.278089 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5293db65-316a-480e-8029-483fb8f0b23d admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:29.326855 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-5293db65-316a-480e-8029-483fb8f0b23d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c9383b84dae04d02bfac17dfa7fc8a25'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:29.334937 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-5293db65-316a-480e-8029-483fb8f0b23d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c9383b84dae04d02bfac17dfa7fc8a25'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:29.348034 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-5293db65-316a-480e-8029-483fb8f0b23d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c9383b84dae04d02bfac17dfa7fc8a25'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:29.348725 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-5293db65-316a-480e-8029-483fb8f0b23d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c9383b84dae04d02bfac17dfa7fc8a25'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:29.355537 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-5293db65-316a-480e-8029-483fb8f0b23d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c9383b84dae04d02bfac17dfa7fc8a25'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:29.362012 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 906/1796] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:36:29 2022] DELETE /identity/v3/users/c9383b84dae04d02bfac17dfa7fc8a25 => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:36:29.366469 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3d76918-2964-4b17-938d-766dde9c5ab7 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:29.367002 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3d76918-2964-4b17-938d-766dde9c5ab7 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:29.367002 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3d76918-2964-4b17-938d-766dde9c5ab7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:29.390197 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-c3d76918-2964-4b17-938d-766dde9c5ab7 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:36:29.391502 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-c3d76918-2964-4b17-938d-766dde9c5ab7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:29.401225 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:36:29.500021 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 891/1797] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:36:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:36:29.509115 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5293db65-316a-480e-8029-483fb8f0b23d admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:29.512925 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-5293db65-316a-480e-8029-483fb8f0b23d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:29.589059 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5293db65-316a-480e-8029-483fb8f0b23d admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:29.594020 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15dd30e8-f8cf-4ccb-9c4d-daebd2e37b99 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:29.594838 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15dd30e8-f8cf-4ccb-9c4d-daebd2e37b99 service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:29.595123 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15dd30e8-f8cf-4ccb-9c4d-daebd2e37b99 service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:29.595409 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15dd30e8-f8cf-4ccb-9c4d-daebd2e37b99 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:29.606533 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15dd30e8-f8cf-4ccb-9c4d-daebd2e37b99 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:29.609179 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15dd30e8-f8cf-4ccb-9c4d-daebd2e37b99 service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:29.625676 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 907/1798] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:36:29 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 117 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:36:29.947573 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3d76918-2964-4b17-938d-766dde9c5ab7 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:29.949023 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-c3d76918-2964-4b17-938d-766dde9c5ab7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:29.963669 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3d76918-2964-4b17-938d-766dde9c5ab7 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:29.969490 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8075946-8565-4f9b-8737-b35e53854cfc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:29.970273 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8075946-8565-4f9b-8737-b35e53854cfc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:29.970560 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8075946-8565-4f9b-8737-b35e53854cfc admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:29.970835 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8075946-8565-4f9b-8737-b35e53854cfc admin admin] PATH_INFO: `/v3/projects/e090fa2d0a824c9c9e6081cc7928ebb8` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:29.972110 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8075946-8565-4f9b-8737-b35e53854cfc admin admin] RBAC: Authorizing `identity:delete_project(project_id=e090fa2d0a824c9c9e6081cc7928ebb8)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:29.974652 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8075946-8565-4f9b-8737-b35e53854cfc admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:30.048969 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-c8075946-8565-4f9b-8737-b35e53854cfc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e090fa2d0a824c9c9e6081cc7928ebb8', 'request_id': 'req-c8075946-8565-4f9b-8737-b35e53854cfc'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:30.064123 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-c8075946-8565-4f9b-8737-b35e53854cfc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e090fa2d0a824c9c9e6081cc7928ebb8', 'request_id': 'req-c8075946-8565-4f9b-8737-b35e53854cfc'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:30.096022 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-c8075946-8565-4f9b-8737-b35e53854cfc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e090fa2d0a824c9c9e6081cc7928ebb8', 'request_id': 'req-c8075946-8565-4f9b-8737-b35e53854cfc'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:30.128307 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-c8075946-8565-4f9b-8737-b35e53854cfc admin admin] The token cache is being invalidate because project e090fa2d0a824c9c9e6081cc7928ebb8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:36:30.131028 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-c8075946-8565-4f9b-8737-b35e53854cfc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:30.133059 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-c8075946-8565-4f9b-8737-b35e53854cfc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:30.140473 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 892/1799] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:36:29 2022] DELETE /identity/v3/projects/e090fa2d0a824c9c9e6081cc7928ebb8 => generated 0 bytes in 189 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:36:30.141017 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15dd30e8-f8cf-4ccb-9c4d-daebd2e37b99 service neutron] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:30.152188 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15dd30e8-f8cf-4ccb-9c4d-daebd2e37b99 service neutron] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:30.190465 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:36:30.256315 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7820a30-c53a-4f35-808a-55c7c69a060f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:30.257025 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7820a30-c53a-4f35-808a-55c7c69a060f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:30.257306 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7820a30-c53a-4f35-808a-55c7c69a060f admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:30.257596 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7820a30-c53a-4f35-808a-55c7c69a060f admin admin] PATH_INFO: `/v3/users/2d429f764c6d4a2e830e9c278cbf9c79` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:30.258816 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7820a30-c53a-4f35-808a-55c7c69a060f admin admin] RBAC: Authorizing `identity:delete_user(user_id=2d429f764c6d4a2e830e9c278cbf9c79)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:30.260934 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7820a30-c53a-4f35-808a-55c7c69a060f admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:30.329794 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-d7820a30-c53a-4f35-808a-55c7c69a060f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2d429f764c6d4a2e830e9c278cbf9c79'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:30.339067 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-d7820a30-c53a-4f35-808a-55c7c69a060f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2d429f764c6d4a2e830e9c278cbf9c79'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:30.352204 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-d7820a30-c53a-4f35-808a-55c7c69a060f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2d429f764c6d4a2e830e9c278cbf9c79'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:30.352924 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-d7820a30-c53a-4f35-808a-55c7c69a060f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2d429f764c6d4a2e830e9c278cbf9c79'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:30.360925 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-d7820a30-c53a-4f35-808a-55c7c69a060f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2d429f764c6d4a2e830e9c278cbf9c79'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:30.368105 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 908/1800] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:36:30 2022] DELETE /identity/v3/users/2d429f764c6d4a2e830e9c278cbf9c79 => generated 0 bytes in 228 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:36:30.739009 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8075946-8565-4f9b-8737-b35e53854cfc admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:30.740556 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-c8075946-8565-4f9b-8737-b35e53854cfc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:30.758643 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8075946-8565-4f9b-8737-b35e53854cfc admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:30.824570 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-906f8af5-3178-43c1-8e05-5da8af57dd5f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:30.825253 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-906f8af5-3178-43c1-8e05-5da8af57dd5f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:30.826193 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-906f8af5-3178-43c1-8e05-5da8af57dd5f admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:30.826483 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-906f8af5-3178-43c1-8e05-5da8af57dd5f admin admin] PATH_INFO: `/v3/projects/b48acfe4b8104d34859cfafc7c538f02` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:30.827637 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-906f8af5-3178-43c1-8e05-5da8af57dd5f admin admin] RBAC: Authorizing `identity:delete_project(project_id=b48acfe4b8104d34859cfafc7c538f02)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:30.829585 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-906f8af5-3178-43c1-8e05-5da8af57dd5f admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:30.870271 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:36:30.877680 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-906f8af5-3178-43c1-8e05-5da8af57dd5f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b48acfe4b8104d34859cfafc7c538f02', 'request_id': 'req-906f8af5-3178-43c1-8e05-5da8af57dd5f'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:30.885076 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-906f8af5-3178-43c1-8e05-5da8af57dd5f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b48acfe4b8104d34859cfafc7c538f02', 'request_id': 'req-906f8af5-3178-43c1-8e05-5da8af57dd5f'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:30.891222 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-906f8af5-3178-43c1-8e05-5da8af57dd5f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b48acfe4b8104d34859cfafc7c538f02', 'request_id': 'req-906f8af5-3178-43c1-8e05-5da8af57dd5f'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:30.902682 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-906f8af5-3178-43c1-8e05-5da8af57dd5f admin admin] The token cache is being invalidate because project b48acfe4b8104d34859cfafc7c538f02 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:36:30.902972 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-906f8af5-3178-43c1-8e05-5da8af57dd5f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:30.904211 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-906f8af5-3178-43c1-8e05-5da8af57dd5f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:30.906100 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 893/1801] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:36:30 2022] DELETE /identity/v3/projects/b48acfe4b8104d34859cfafc7c538f02 => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:36:30.910987 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83c737b7-62e5-4704-bf5b-3f6f4bc44399 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:30.911399 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83c737b7-62e5-4704-bf5b-3f6f4bc44399 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:30.911825 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83c737b7-62e5-4704-bf5b-3f6f4bc44399 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:30.931952 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-83c737b7-62e5-4704-bf5b-3f6f4bc44399 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:36:30.933291 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-83c737b7-62e5-4704-bf5b-3f6f4bc44399 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:30.999823 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 909/1802] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:36:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:36:31.008123 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc11b8e4-f99e-4420-979b-388fe3b5caed None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:31.008404 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc11b8e4-f99e-4420-979b-388fe3b5caed None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:31.008917 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc11b8e4-f99e-4420-979b-388fe3b5caed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:31.029977 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-dc11b8e4-f99e-4420-979b-388fe3b5caed None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:36:31.031273 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-dc11b8e4-f99e-4420-979b-388fe3b5caed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:31.043008 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 894/1803] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:36:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:36:31.044673 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83c737b7-62e5-4704-bf5b-3f6f4bc44399 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:31.054847 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83c737b7-62e5-4704-bf5b-3f6f4bc44399 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:31.059580 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f86cb5e8-aacc-4bac-8320-1ca5f3a41158 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:31.061171 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f86cb5e8-aacc-4bac-8320-1ca5f3a41158 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:31.061445 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f86cb5e8-aacc-4bac-8320-1ca5f3a41158 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:31.061714 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f86cb5e8-aacc-4bac-8320-1ca5f3a41158 admin admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:31.062360 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f86cb5e8-aacc-4bac-8320-1ca5f3a41158 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:36:31.062664 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f86cb5e8-aacc-4bac-8320-1ca5f3a41158 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:31.064270 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f86cb5e8-aacc-4bac-8320-1ca5f3a41158 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:31.077227 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 910/1804] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Sep 25 04:36:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:36:31.082463 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc11b8e4-f99e-4420-979b-388fe3b5caed None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:31.110375 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc11b8e4-f99e-4420-979b-388fe3b5caed None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:31.115877 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6e143a2-fa64-4bf1-8369-03819bf30f6a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:31.116789 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6e143a2-fa64-4bf1-8369-03819bf30f6a admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:31.117253 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6e143a2-fa64-4bf1-8369-03819bf30f6a admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:31.118360 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6e143a2-fa64-4bf1-8369-03819bf30f6a admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:31.119300 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6e143a2-fa64-4bf1-8369-03819bf30f6a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:31.121603 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6e143a2-fa64-4bf1-8369-03819bf30f6a admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:31.157422 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 895/1805] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:36:31 2022] POST /identity/v3/projects => generated 363 bytes in 76 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:36:31.158862 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f86cb5e8-aacc-4bac-8320-1ca5f3a41158 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:31.175460 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f86cb5e8-aacc-4bac-8320-1ca5f3a41158 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:31.207645 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:36:31.275573 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b004109-e791-43e1-890c-5719df8901ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:31.276289 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b004109-e791-43e1-890c-5719df8901ed admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:31.277469 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b004109-e791-43e1-890c-5719df8901ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:31.277761 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b004109-e791-43e1-890c-5719df8901ed admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:31.278994 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b004109-e791-43e1-890c-5719df8901ed admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:31.281029 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b004109-e791-43e1-890c-5719df8901ed admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:31.308891 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 911/1806] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:36:31 2022] POST /identity/v3/users => generated 282 bytes in 148 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:36:31.311261 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6e143a2-fa64-4bf1-8369-03819bf30f6a admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:31.322273 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6e143a2-fa64-4bf1-8369-03819bf30f6a admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:31.328017 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-640ca67a-0850-4ecd-85bd-5f1c84ddcd97 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:31.329300 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-640ca67a-0850-4ecd-85bd-5f1c84ddcd97 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:31.329593 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-640ca67a-0850-4ecd-85bd-5f1c84ddcd97 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:31.329876 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-640ca67a-0850-4ecd-85bd-5f1c84ddcd97 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:31.330736 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-640ca67a-0850-4ecd-85bd-5f1c84ddcd97 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:31.332668 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-640ca67a-0850-4ecd-85bd-5f1c84ddcd97 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:31.344781 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 896/1807] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:36:31 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:36:31.347688 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b004109-e791-43e1-890c-5719df8901ed admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:31.357744 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b004109-e791-43e1-890c-5719df8901ed admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:31.362362 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b96a92b-e8cb-4c5c-a920-75c1b9ed9694 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:31.362958 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b96a92b-e8cb-4c5c-a920-75c1b9ed9694 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:31.363251 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b96a92b-e8cb-4c5c-a920-75c1b9ed9694 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:31.363523 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b96a92b-e8cb-4c5c-a920-75c1b9ed9694 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:31.364157 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b96a92b-e8cb-4c5c-a920-75c1b9ed9694 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:31.365882 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b96a92b-e8cb-4c5c-a920-75c1b9ed9694 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:31.377465 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 912/1808] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:36:31 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:36:31.381696 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-640ca67a-0850-4ecd-85bd-5f1c84ddcd97 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:31.392392 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-640ca67a-0850-4ecd-85bd-5f1c84ddcd97 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:31.397480 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afb14d0d-86ce-4d2b-a01a-8748887ade6d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:31.398132 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afb14d0d-86ce-4d2b-a01a-8748887ade6d admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:31.398419 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afb14d0d-86ce-4d2b-a01a-8748887ade6d admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:31.398809 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afb14d0d-86ce-4d2b-a01a-8748887ade6d admin admin] PATH_INFO: `/v3/projects/5b17c9aa4e2c49aa923d78d6ce37e9bf/users/2acef52371214f64ba1fa31afe76dfbf/roles/8f6cc574056c4278bc9778f14dc4b411` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:31.415436 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afb14d0d-86ce-4d2b-a01a-8748887ade6d admin admin] RBAC: Authorizing `identity:create_grant(project_id=5b17c9aa4e2c49aa923d78d6ce37e9bf, user_id=2acef52371214f64ba1fa31afe76dfbf, role_id=8f6cc574056c4278bc9778f14dc4b411)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:31.418154 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afb14d0d-86ce-4d2b-a01a-8748887ade6d admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:31.430703 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 897/1809] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:36:31 2022] PUT /identity/v3/projects/5b17c9aa4e2c49aa923d78d6ce37e9bf/users/2acef52371214f64ba1fa31afe76dfbf/roles/8f6cc574056c4278bc9778f14dc4b411 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:36:31.442804 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ac9031e-221b-4692-b760-21a373b84b18 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:31.443095 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ac9031e-221b-4692-b760-21a373b84b18 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:31.443372 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ac9031e-221b-4692-b760-21a373b84b18 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:31.475628 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-4ac9031e-221b-4692-b760-21a373b84b18 None None] MFA Rules not processed for user `2acef52371214f64ba1fa31afe76dfbf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:36:31.476905 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-4ac9031e-221b-4692-b760-21a373b84b18 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:31.555498 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 913/1810] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:36:31 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:36:31.560975 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afb14d0d-86ce-4d2b-a01a-8748887ade6d admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:31.628927 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afb14d0d-86ce-4d2b-a01a-8748887ade6d admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:31.633984 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc1017ef-ac81-450b-a9c0-eb8d9d0fc356 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:31.634607 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc1017ef-ac81-450b-a9c0-eb8d9d0fc356 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:31.634913 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc1017ef-ac81-450b-a9c0-eb8d9d0fc356 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:31.635299 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc1017ef-ac81-450b-a9c0-eb8d9d0fc356 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:31.635837 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc1017ef-ac81-450b-a9c0-eb8d9d0fc356 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:31.637880 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc1017ef-ac81-450b-a9c0-eb8d9d0fc356 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:31.668818 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 898/1811] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:36:31 2022] POST /identity/v3/projects => generated 365 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:36:31.673052 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ac9031e-221b-4692-b760-21a373b84b18 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:31.684097 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ac9031e-221b-4692-b760-21a373b84b18 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:31.745038 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91b62d7e-52c8-4ca5-96a0-1c40da2e753f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:31.745742 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91b62d7e-52c8-4ca5-96a0-1c40da2e753f admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:31.746164 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91b62d7e-52c8-4ca5-96a0-1c40da2e753f admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:31.746342 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91b62d7e-52c8-4ca5-96a0-1c40da2e753f admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:31.746992 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91b62d7e-52c8-4ca5-96a0-1c40da2e753f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:31.749853 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91b62d7e-52c8-4ca5-96a0-1c40da2e753f admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:31.776674 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 914/1812] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:36:31 2022] POST /identity/v3/users => generated 289 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:36:31.778900 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc1017ef-ac81-450b-a9c0-eb8d9d0fc356 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:31.789541 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc1017ef-ac81-450b-a9c0-eb8d9d0fc356 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:31.795155 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69de49ae-d092-44bb-98b2-bd2015de78e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:31.795953 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69de49ae-d092-44bb-98b2-bd2015de78e7 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:31.796237 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69de49ae-d092-44bb-98b2-bd2015de78e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:31.796515 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69de49ae-d092-44bb-98b2-bd2015de78e7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:31.797235 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69de49ae-d092-44bb-98b2-bd2015de78e7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:31.798935 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69de49ae-d092-44bb-98b2-bd2015de78e7 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:31.813678 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 899/1813] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:36:31 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:36:31.814848 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91b62d7e-52c8-4ca5-96a0-1c40da2e753f admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:31.825040 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91b62d7e-52c8-4ca5-96a0-1c40da2e753f admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:31.831140 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d84408bb-1047-4110-89bb-75802b2c8776 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:31.831404 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d84408bb-1047-4110-89bb-75802b2c8776 admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:31.831688 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d84408bb-1047-4110-89bb-75802b2c8776 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:31.831958 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d84408bb-1047-4110-89bb-75802b2c8776 admin admin] PATH_INFO: `/v3/projects/6bafdf52ef5c4793becbab3a1b69cb08/users/cc73e75e4283421381df42b95dc0f593/roles/c51fe7c5ad804040879acfb54269455f` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:31.847960 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d84408bb-1047-4110-89bb-75802b2c8776 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6bafdf52ef5c4793becbab3a1b69cb08, user_id=cc73e75e4283421381df42b95dc0f593, role_id=c51fe7c5ad804040879acfb54269455f)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:31.850646 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d84408bb-1047-4110-89bb-75802b2c8776 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:31.852872 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:36:31.867965 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 915/1814] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:36:31 2022] PUT /identity/v3/projects/6bafdf52ef5c4793becbab3a1b69cb08/users/cc73e75e4283421381df42b95dc0f593/roles/c51fe7c5ad804040879acfb54269455f => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 04:36:31.871803 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69de49ae-d092-44bb-98b2-bd2015de78e7 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:31.882636 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69de49ae-d092-44bb-98b2-bd2015de78e7 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:31.918080 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=92,keystone:DELETE=4,keystone:INSERT=4 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:36:31.944082 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c11f02b-693a-4bea-a7ab-943f0ad78ee0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:31.944766 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c11f02b-693a-4bea-a7ab-943f0ad78ee0 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:31.945064 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c11f02b-693a-4bea-a7ab-943f0ad78ee0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:31.945344 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c11f02b-693a-4bea-a7ab-943f0ad78ee0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:31.945990 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c11f02b-693a-4bea-a7ab-943f0ad78ee0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:31.947634 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c11f02b-693a-4bea-a7ab-943f0ad78ee0 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:31.962734 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 900/1815] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:36:31 2022] GET /identity/v3/roles => generated 1420 bytes in 92 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:36:31.967573 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d84408bb-1047-4110-89bb-75802b2c8776 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:31.978051 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d84408bb-1047-4110-89bb-75802b2c8776 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:31.983685 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-155af4b7-c795-4c84-bed8-6e1da9235ee2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:31.984395 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-155af4b7-c795-4c84-bed8-6e1da9235ee2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:31.984710 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-155af4b7-c795-4c84-bed8-6e1da9235ee2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:31.985014 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-155af4b7-c795-4c84-bed8-6e1da9235ee2 admin admin] PATH_INFO: `/v3/projects/6bafdf52ef5c4793becbab3a1b69cb08/users/cc73e75e4283421381df42b95dc0f593/roles/c51fe7c5ad804040879acfb54269455f` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:31.986930 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-155af4b7-c795-4c84-bed8-6e1da9235ee2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6bafdf52ef5c4793becbab3a1b69cb08, user_id=cc73e75e4283421381df42b95dc0f593, role_id=c51fe7c5ad804040879acfb54269455f)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:31.989680 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-155af4b7-c795-4c84-bed8-6e1da9235ee2 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:32.000046 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 916/1816] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:36:31 2022] PUT /identity/v3/projects/6bafdf52ef5c4793becbab3a1b69cb08/users/cc73e75e4283421381df42b95dc0f593/roles/c51fe7c5ad804040879acfb54269455f => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 04:36:32.010468 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae765df3-000f-4b98-aa57-82b8d5e342b0 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:32.010773 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae765df3-000f-4b98-aa57-82b8d5e342b0 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:32.011045 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae765df3-000f-4b98-aa57-82b8d5e342b0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:32.039014 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-ae765df3-000f-4b98-aa57-82b8d5e342b0 None None] MFA Rules not processed for user `cc73e75e4283421381df42b95dc0f593`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:36:32.040210 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-ae765df3-000f-4b98-aa57-82b8d5e342b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:32.108585 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 901/1817] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:36:32 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:36:32.120024 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-155af4b7-c795-4c84-bed8-6e1da9235ee2 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:32.121954 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-155af4b7-c795-4c84-bed8-6e1da9235ee2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:32.188541 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-155af4b7-c795-4c84-bed8-6e1da9235ee2 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:32.192189 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ffa1668-68c6-4432-a77d-153a2c1355a9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ffbc136c77b84e8c87128f7247f4ec03', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:32.192839 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ffa1668-68c6-4432-a77d-153a2c1355a9 service cinder] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:32.193121 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ffa1668-68c6-4432-a77d-153a2c1355a9 service cinder] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:32.193385 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ffa1668-68c6-4432-a77d-153a2c1355a9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:32.202719 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ffa1668-68c6-4432-a77d-153a2c1355a9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:32.204529 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ffa1668-68c6-4432-a77d-153a2c1355a9 service cinder] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:32.215431 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 917/1818] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:36:32 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:36:32.331836 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae765df3-000f-4b98-aa57-82b8d5e342b0 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:32.342656 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae765df3-000f-4b98-aa57-82b8d5e342b0 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:32.346264 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c0a63d9-35d9-48be-9f5f-43477dd3c118 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ffbc136c77b84e8c87128f7247f4ec03', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:32.346873 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c0a63d9-35d9-48be-9f5f-43477dd3c118 service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:32.347150 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c0a63d9-35d9-48be-9f5f-43477dd3c118 service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:32.347452 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c0a63d9-35d9-48be-9f5f-43477dd3c118 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:32.413474 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c0a63d9-35d9-48be-9f5f-43477dd3c118 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:32.415016 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c0a63d9-35d9-48be-9f5f-43477dd3c118 service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:32.425055 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 902/1819] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:36:32 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:36:34.813484 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce11d93-1f91-4715-9792-24ba6b80a965 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:34.814654 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce11d93-1f91-4715-9792-24ba6b80a965 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:34.815183 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce11d93-1f91-4715-9792-24ba6b80a965 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:34.817182 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 918/1820] 10.222.0.25 () {54 vars in 921 bytes} [Sun Sep 25 04:36:34 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:36:34.832950 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b62d3c80-943e-435e-a64e-f9699ff50e6d None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:34.833890 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b62d3c80-943e-435e-a64e-f9699ff50e6d None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:34.834314 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b62d3c80-943e-435e-a64e-f9699ff50e6d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:34.875497 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-b62d3c80-943e-435e-a64e-f9699ff50e6d None None] MFA Rules not processed for user `e703b42c53974daaa608b0ccc3f018b1`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:36:34.876961 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-b62d3c80-943e-435e-a64e-f9699ff50e6d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:34.987977 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:36:35.009948 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 903/1821] 10.222.0.25 () {58 vars in 1018 bytes} [Sun Sep 25 04:36:34 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:36:35.093365 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ce11d93-1f91-4715-9792-24ba6b80a965 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:35.095309 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-6ce11d93-1f91-4715-9792-24ba6b80a965 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:35.114832 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ce11d93-1f91-4715-9792-24ba6b80a965 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:35.124804 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c0b6b6e-594d-40e6-aaed-cc38c5e1ff45 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e703b42c53974daaa608b0ccc3f018b1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'ResellerAdmin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:35.125486 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c0b6b6e-594d-40e6-aaed-cc38c5e1ff45 service nova] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:35.125832 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c0b6b6e-594d-40e6-aaed-cc38c5e1ff45 service nova] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:35.126165 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c0b6b6e-594d-40e6-aaed-cc38c5e1ff45 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:35.135975 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c0b6b6e-594d-40e6-aaed-cc38c5e1ff45 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:35.137514 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c0b6b6e-594d-40e6-aaed-cc38c5e1ff45 service nova] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:35.148543 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 919/1822] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:36:35 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 56 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:36:42.650819 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b62d3c80-943e-435e-a64e-f9699ff50e6d None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:42.743800 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b62d3c80-943e-435e-a64e-f9699ff50e6d None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:42.749971 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0f92b19-2648-4fe6-bd93-08c8e75402f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:42.750907 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f92b19-2648-4fe6-bd93-08c8e75402f9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:42.751421 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f92b19-2648-4fe6-bd93-08c8e75402f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:42.751911 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f92b19-2648-4fe6-bd93-08c8e75402f9 admin admin] PATH_INFO: `/v3/users/2acef52371214f64ba1fa31afe76dfbf` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:42.753596 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0f92b19-2648-4fe6-bd93-08c8e75402f9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2acef52371214f64ba1fa31afe76dfbf)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:42.755803 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0f92b19-2648-4fe6-bd93-08c8e75402f9 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:42.823800 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-f0f92b19-2648-4fe6-bd93-08c8e75402f9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2acef52371214f64ba1fa31afe76dfbf'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:42.840209 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-f0f92b19-2648-4fe6-bd93-08c8e75402f9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2acef52371214f64ba1fa31afe76dfbf'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:42.874179 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-f0f92b19-2648-4fe6-bd93-08c8e75402f9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2acef52371214f64ba1fa31afe76dfbf'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:42.879405 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-f0f92b19-2648-4fe6-bd93-08c8e75402f9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2acef52371214f64ba1fa31afe76dfbf'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:42.898318 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-f0f92b19-2648-4fe6-bd93-08c8e75402f9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2acef52371214f64ba1fa31afe76dfbf'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:42.917778 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 904/1823] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:36:42 2022] DELETE /identity/v3/users/2acef52371214f64ba1fa31afe76dfbf => generated 0 bytes in 268 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:36:42.923315 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1307c6bb-7f65-4ce8-9dd4-6fc8dfc31b36 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:42.923866 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1307c6bb-7f65-4ce8-9dd4-6fc8dfc31b36 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:42.923950 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1307c6bb-7f65-4ce8-9dd4-6fc8dfc31b36 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:42.949934 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-1307c6bb-7f65-4ce8-9dd4-6fc8dfc31b36 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:36:42.952012 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-1307c6bb-7f65-4ce8-9dd4-6fc8dfc31b36 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:43.062980 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 920/1824] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:36:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:36:43.118564 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0f92b19-2648-4fe6-bd93-08c8e75402f9 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:43.120235 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-f0f92b19-2648-4fe6-bd93-08c8e75402f9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:43.293672 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0f92b19-2648-4fe6-bd93-08c8e75402f9 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:43.310484 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0b096a5-d771-48a6-a345-1f967d7422f6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:43.312463 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0b096a5-d771-48a6-a345-1f967d7422f6 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:43.313546 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0b096a5-d771-48a6-a345-1f967d7422f6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:43.315202 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0b096a5-d771-48a6-a345-1f967d7422f6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:43.341134 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0b096a5-d771-48a6-a345-1f967d7422f6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:43.344933 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0b096a5-d771-48a6-a345-1f967d7422f6 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:43.368822 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 905/1825] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:36:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 251 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:36:43.818341 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1307c6bb-7f65-4ce8-9dd4-6fc8dfc31b36 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:43.820021 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-1307c6bb-7f65-4ce8-9dd4-6fc8dfc31b36 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:43.876711 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1307c6bb-7f65-4ce8-9dd4-6fc8dfc31b36 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:43.884574 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a0f6b27-dadc-4a90-a1d5-d89d64829dc5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:43.885455 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a0f6b27-dadc-4a90-a1d5-d89d64829dc5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:43.886026 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a0f6b27-dadc-4a90-a1d5-d89d64829dc5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:43.886319 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a0f6b27-dadc-4a90-a1d5-d89d64829dc5 admin admin] PATH_INFO: `/v3/users/71b36f0919a7427c87ce7cd76e85e4fd` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:43.888581 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a0f6b27-dadc-4a90-a1d5-d89d64829dc5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=71b36f0919a7427c87ce7cd76e85e4fd)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:43.891682 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a0f6b27-dadc-4a90-a1d5-d89d64829dc5 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:43.998280 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-3a0f6b27-dadc-4a90-a1d5-d89d64829dc5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '71b36f0919a7427c87ce7cd76e85e4fd'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:44.008368 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-3a0f6b27-dadc-4a90-a1d5-d89d64829dc5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '71b36f0919a7427c87ce7cd76e85e4fd'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:44.023781 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:INSERT=3,keystone:SELECT=88,keystone:DELETE=9 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:36:44.038797 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-3a0f6b27-dadc-4a90-a1d5-d89d64829dc5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '71b36f0919a7427c87ce7cd76e85e4fd'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:44.042321 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-3a0f6b27-dadc-4a90-a1d5-d89d64829dc5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '71b36f0919a7427c87ce7cd76e85e4fd'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:44.057174 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-3a0f6b27-dadc-4a90-a1d5-d89d64829dc5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '71b36f0919a7427c87ce7cd76e85e4fd'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:44.074110 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 921/1826] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:36:43 2022] DELETE /identity/v3/users/71b36f0919a7427c87ce7cd76e85e4fd => generated 0 bytes in 256 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:36:44.080168 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7ce4202-2f30-4f93-b63e-4339472bf2a5 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:44.081231 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7ce4202-2f30-4f93-b63e-4339472bf2a5 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:44.081656 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7ce4202-2f30-4f93-b63e-4339472bf2a5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:44.116708 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-e7ce4202-2f30-4f93-b63e-4339472bf2a5 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:36:44.118011 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-e7ce4202-2f30-4f93-b63e-4339472bf2a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:44.123970 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:36:44.243973 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 906/1827] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:36:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:36:44.257677 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a0f6b27-dadc-4a90-a1d5-d89d64829dc5 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:44.259866 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-3a0f6b27-dadc-4a90-a1d5-d89d64829dc5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:44.353461 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a0f6b27-dadc-4a90-a1d5-d89d64829dc5 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:44.358439 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0359bfd-d743-40c6-8882-75f62bef731d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:44.359328 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0359bfd-d743-40c6-8882-75f62bef731d service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:44.359791 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0359bfd-d743-40c6-8882-75f62bef731d service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:44.360444 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0359bfd-d743-40c6-8882-75f62bef731d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:44.384927 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0359bfd-d743-40c6-8882-75f62bef731d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:44.389006 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0359bfd-d743-40c6-8882-75f62bef731d service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:44.420056 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 922/1828] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:36:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 163 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:36:45.051919 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7ce4202-2f30-4f93-b63e-4339472bf2a5 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:45.053399 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-e7ce4202-2f30-4f93-b63e-4339472bf2a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:45.066321 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7ce4202-2f30-4f93-b63e-4339472bf2a5 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:45.080978 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88b782b9-2f2c-4d0b-93b1-ddcdf5719d3c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:45.082704 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88b782b9-2f2c-4d0b-93b1-ddcdf5719d3c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:45.083326 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88b782b9-2f2c-4d0b-93b1-ddcdf5719d3c admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:45.083755 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88b782b9-2f2c-4d0b-93b1-ddcdf5719d3c admin admin] PATH_INFO: `/v3/projects/957087b9343a41ddbdb7c28fda2de2f3` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:45.087178 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88b782b9-2f2c-4d0b-93b1-ddcdf5719d3c admin admin] RBAC: Authorizing `identity:delete_project(project_id=957087b9343a41ddbdb7c28fda2de2f3)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:45.096876 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88b782b9-2f2c-4d0b-93b1-ddcdf5719d3c admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:45.129287 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0359bfd-d743-40c6-8882-75f62bef731d service neutron] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:45.131726 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-a0359bfd-d743-40c6-8882-75f62bef731d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:45.153684 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0359bfd-d743-40c6-8882-75f62bef731d service neutron] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:45.184194 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a09e860-08cb-4e82-bb60-ce1ecbf76d73 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:45.185550 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a09e860-08cb-4e82-bb60-ce1ecbf76d73 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:45.185987 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a09e860-08cb-4e82-bb60-ce1ecbf76d73 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:45.186486 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a09e860-08cb-4e82-bb60-ce1ecbf76d73 admin admin] PATH_INFO: `/v3/projects/5b17c9aa4e2c49aa923d78d6ce37e9bf` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:45.190758 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a09e860-08cb-4e82-bb60-ce1ecbf76d73 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5b17c9aa4e2c49aa923d78d6ce37e9bf)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:45.196571 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a09e860-08cb-4e82-bb60-ce1ecbf76d73 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:45.210457 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-88b782b9-2f2c-4d0b-93b1-ddcdf5719d3c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '957087b9343a41ddbdb7c28fda2de2f3', 'request_id': 'req-88b782b9-2f2c-4d0b-93b1-ddcdf5719d3c'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:45.235287 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-88b782b9-2f2c-4d0b-93b1-ddcdf5719d3c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '957087b9343a41ddbdb7c28fda2de2f3', 'request_id': 'req-88b782b9-2f2c-4d0b-93b1-ddcdf5719d3c'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:45.245763 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-88b782b9-2f2c-4d0b-93b1-ddcdf5719d3c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '957087b9343a41ddbdb7c28fda2de2f3', 'request_id': 'req-88b782b9-2f2c-4d0b-93b1-ddcdf5719d3c'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:45.261193 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-88b782b9-2f2c-4d0b-93b1-ddcdf5719d3c admin admin] The token cache is being invalidate because project 957087b9343a41ddbdb7c28fda2de2f3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:36:45.261824 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-88b782b9-2f2c-4d0b-93b1-ddcdf5719d3c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:45.268208 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-88b782b9-2f2c-4d0b-93b1-ddcdf5719d3c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:45.273860 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 907/1829] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:36:45 2022] DELETE /identity/v3/projects/957087b9343a41ddbdb7c28fda2de2f3 => generated 0 bytes in 222 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:36:45.286001 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88b782b9-2f2c-4d0b-93b1-ddcdf5719d3c admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:45.288410 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-88b782b9-2f2c-4d0b-93b1-ddcdf5719d3c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:45.336942 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-4a09e860-08cb-4e82-bb60-ce1ecbf76d73 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5b17c9aa4e2c49aa923d78d6ce37e9bf', 'request_id': 'req-4a09e860-08cb-4e82-bb60-ce1ecbf76d73'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:45.364972 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-4a09e860-08cb-4e82-bb60-ce1ecbf76d73 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5b17c9aa4e2c49aa923d78d6ce37e9bf', 'request_id': 'req-4a09e860-08cb-4e82-bb60-ce1ecbf76d73'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:45.380666 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-4a09e860-08cb-4e82-bb60-ce1ecbf76d73 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5b17c9aa4e2c49aa923d78d6ce37e9bf', 'request_id': 'req-4a09e860-08cb-4e82-bb60-ce1ecbf76d73'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:45.412161 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-4a09e860-08cb-4e82-bb60-ce1ecbf76d73 admin admin] The token cache is being invalidate because project 5b17c9aa4e2c49aa923d78d6ce37e9bf was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81434) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:36:45.412907 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-4a09e860-08cb-4e82-bb60-ce1ecbf76d73 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:45.420025 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-4a09e860-08cb-4e82-bb60-ce1ecbf76d73 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:45.422426 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 923/1830] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:36:45 2022] DELETE /identity/v3/projects/5b17c9aa4e2c49aa923d78d6ce37e9bf => generated 0 bytes in 294 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:36:45.427179 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a09e860-08cb-4e82-bb60-ce1ecbf76d73 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:45.430855 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-4a09e860-08cb-4e82-bb60-ce1ecbf76d73 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:45.500533 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88b782b9-2f2c-4d0b-93b1-ddcdf5719d3c admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:45.607548 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:36:45.616902 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a09e860-08cb-4e82-bb60-ce1ecbf76d73 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:45.623448 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cecaa5f-e66a-4f35-b60b-b120816a5e69 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:45.624260 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cecaa5f-e66a-4f35-b60b-b120816a5e69 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:45.625432 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cecaa5f-e66a-4f35-b60b-b120816a5e69 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:45.625619 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cecaa5f-e66a-4f35-b60b-b120816a5e69 admin admin] PATH_INFO: `/v3/users/cc73e75e4283421381df42b95dc0f593` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:45.630082 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cecaa5f-e66a-4f35-b60b-b120816a5e69 admin admin] RBAC: Authorizing `identity:delete_user(user_id=cc73e75e4283421381df42b95dc0f593)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:45.632840 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cecaa5f-e66a-4f35-b60b-b120816a5e69 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:45.633492 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaf5fd28-47be-400b-99f2-6266f0c001ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:45.634838 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf5fd28-47be-400b-99f2-6266f0c001ad admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:45.635085 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf5fd28-47be-400b-99f2-6266f0c001ad admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:45.635402 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf5fd28-47be-400b-99f2-6266f0c001ad admin admin] PATH_INFO: `/v3/users/8f57caa464e6420ebe1d6c766edc5f5c` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:45.637190 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaf5fd28-47be-400b-99f2-6266f0c001ad admin admin] RBAC: Authorizing `identity:delete_user(user_id=8f57caa464e6420ebe1d6c766edc5f5c)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:45.639742 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaf5fd28-47be-400b-99f2-6266f0c001ad admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:45.647233 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:36:45.705710 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-0cecaa5f-e66a-4f35-b60b-b120816a5e69 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cc73e75e4283421381df42b95dc0f593'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:45.714733 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-eaf5fd28-47be-400b-99f2-6266f0c001ad admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8f57caa464e6420ebe1d6c766edc5f5c'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:45.716482 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-0cecaa5f-e66a-4f35-b60b-b120816a5e69 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cc73e75e4283421381df42b95dc0f593'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:45.723829 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-eaf5fd28-47be-400b-99f2-6266f0c001ad admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8f57caa464e6420ebe1d6c766edc5f5c'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:45.733828 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-0cecaa5f-e66a-4f35-b60b-b120816a5e69 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cc73e75e4283421381df42b95dc0f593'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:45.734801 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-0cecaa5f-e66a-4f35-b60b-b120816a5e69 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cc73e75e4283421381df42b95dc0f593'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:45.742241 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-0cecaa5f-e66a-4f35-b60b-b120816a5e69 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cc73e75e4283421381df42b95dc0f593'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:45.746414 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-eaf5fd28-47be-400b-99f2-6266f0c001ad admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8f57caa464e6420ebe1d6c766edc5f5c'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:45.747841 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-eaf5fd28-47be-400b-99f2-6266f0c001ad admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8f57caa464e6420ebe1d6c766edc5f5c'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:45.749361 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 924/1831] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:36:45 2022] DELETE /identity/v3/users/cc73e75e4283421381df42b95dc0f593 => generated 0 bytes in 322 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:36:45.750395 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cecaa5f-e66a-4f35-b60b-b120816a5e69 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:45.751798 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-0cecaa5f-e66a-4f35-b60b-b120816a5e69 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:45.756095 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-eaf5fd28-47be-400b-99f2-6266f0c001ad admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8f57caa464e6420ebe1d6c766edc5f5c'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:45.765241 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 908/1832] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:36:45 2022] DELETE /identity/v3/users/8f57caa464e6420ebe1d6c766edc5f5c => generated 0 bytes in 487 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:36:45.848818 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cecaa5f-e66a-4f35-b60b-b120816a5e69 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:45.944454 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-868ff636-39fb-4283-b835-8b717864075f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:45.946565 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-868ff636-39fb-4283-b835-8b717864075f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:45.946565 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-868ff636-39fb-4283-b835-8b717864075f admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:45.946565 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-868ff636-39fb-4283-b835-8b717864075f admin admin] PATH_INFO: `/v3/users/60ff17baa7ca44c5b444af1ad8565e33` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:45.947191 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-868ff636-39fb-4283-b835-8b717864075f admin admin] RBAC: Authorizing `identity:delete_user(user_id=60ff17baa7ca44c5b444af1ad8565e33)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:45.949253 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-868ff636-39fb-4283-b835-8b717864075f admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:45.993951 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:DELETE=15,keystone:SELECT=84,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:36:46.006675 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-868ff636-39fb-4283-b835-8b717864075f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '60ff17baa7ca44c5b444af1ad8565e33'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:46.017721 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-868ff636-39fb-4283-b835-8b717864075f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '60ff17baa7ca44c5b444af1ad8565e33'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:46.053681 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-868ff636-39fb-4283-b835-8b717864075f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '60ff17baa7ca44c5b444af1ad8565e33'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:46.054943 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-868ff636-39fb-4283-b835-8b717864075f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '60ff17baa7ca44c5b444af1ad8565e33'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:46.069851 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-868ff636-39fb-4283-b835-8b717864075f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '60ff17baa7ca44c5b444af1ad8565e33'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:46.077154 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 925/1833] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:36:45 2022] DELETE /identity/v3/users/60ff17baa7ca44c5b444af1ad8565e33 => generated 0 bytes in 327 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:36:46.081830 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cabbb3e-d527-4050-a071-5cff0aa6f02c None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:46.082553 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cabbb3e-d527-4050-a071-5cff0aa6f02c None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:46.082632 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cabbb3e-d527-4050-a071-5cff0aa6f02c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:46.115463 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-3cabbb3e-d527-4050-a071-5cff0aa6f02c None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:36:46.118135 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-3cabbb3e-d527-4050-a071-5cff0aa6f02c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:46.209119 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 909/1834] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:36:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 04:36:46.223260 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-868ff636-39fb-4283-b835-8b717864075f admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:46.224873 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-868ff636-39fb-4283-b835-8b717864075f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:46.293875 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-868ff636-39fb-4283-b835-8b717864075f admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:46.297911 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c7ae414-9fbd-410f-8a47-ad536e207f80 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:46.298539 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c7ae414-9fbd-410f-8a47-ad536e207f80 service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:46.298824 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c7ae414-9fbd-410f-8a47-ad536e207f80 service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:46.299101 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c7ae414-9fbd-410f-8a47-ad536e207f80 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:46.312476 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c7ae414-9fbd-410f-8a47-ad536e207f80 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:46.314563 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c7ae414-9fbd-410f-8a47-ad536e207f80 service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:46.330887 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 926/1835] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:36:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:36:46.639205 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cabbb3e-d527-4050-a071-5cff0aa6f02c None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:46.640691 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-3cabbb3e-d527-4050-a071-5cff0aa6f02c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:46.655323 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cabbb3e-d527-4050-a071-5cff0aa6f02c None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:46.663269 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daa8fdbb-4a2c-4d4d-b1c9-7b91232bb9b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:46.663829 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa8fdbb-4a2c-4d4d-b1c9-7b91232bb9b0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:46.664129 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa8fdbb-4a2c-4d4d-b1c9-7b91232bb9b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:46.664442 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa8fdbb-4a2c-4d4d-b1c9-7b91232bb9b0 admin admin] PATH_INFO: `/v3/projects/14c065d1976144cc9c960e2968da9c14` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:46.665610 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daa8fdbb-4a2c-4d4d-b1c9-7b91232bb9b0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=14c065d1976144cc9c960e2968da9c14)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:46.668089 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daa8fdbb-4a2c-4d4d-b1c9-7b91232bb9b0 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:46.785276 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c7ae414-9fbd-410f-8a47-ad536e207f80 service neutron] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:46.787494 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-4c7ae414-9fbd-410f-8a47-ad536e207f80 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:46.806369 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-daa8fdbb-4a2c-4d4d-b1c9-7b91232bb9b0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '14c065d1976144cc9c960e2968da9c14', 'request_id': 'req-daa8fdbb-4a2c-4d4d-b1c9-7b91232bb9b0'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:46.810580 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c7ae414-9fbd-410f-8a47-ad536e207f80 service neutron] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:46.821200 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-daa8fdbb-4a2c-4d4d-b1c9-7b91232bb9b0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '14c065d1976144cc9c960e2968da9c14', 'request_id': 'req-daa8fdbb-4a2c-4d4d-b1c9-7b91232bb9b0'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:46.823674 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=83,keystone:DELETE=15,keystone:INSERT=2 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:36:46.837996 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-daa8fdbb-4a2c-4d4d-b1c9-7b91232bb9b0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '14c065d1976144cc9c960e2968da9c14', 'request_id': 'req-daa8fdbb-4a2c-4d4d-b1c9-7b91232bb9b0'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:46.854044 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-daa8fdbb-4a2c-4d4d-b1c9-7b91232bb9b0 admin admin] The token cache is being invalidate because project 14c065d1976144cc9c960e2968da9c14 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:36:46.855163 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-daa8fdbb-4a2c-4d4d-b1c9-7b91232bb9b0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:46.857135 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-daa8fdbb-4a2c-4d4d-b1c9-7b91232bb9b0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:46.859824 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 910/1836] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:36:46 2022] DELETE /identity/v3/projects/14c065d1976144cc9c960e2968da9c14 => generated 0 bytes in 222 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:36:46.864600 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d7c21bb-af19-4fdf-b346-3f450d979bce None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:46.865517 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d7c21bb-af19-4fdf-b346-3f450d979bce None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:46.865873 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d7c21bb-af19-4fdf-b346-3f450d979bce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:46.908321 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-2d7c21bb-af19-4fdf-b346-3f450d979bce None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:36:46.910225 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-166290b3-6ce4-4c49-b665-a1639169f83d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:46.910878 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-166290b3-6ce4-4c49-b665-a1639169f83d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:46.911251 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-166290b3-6ce4-4c49-b665-a1639169f83d admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:46.911782 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-2d7c21bb-af19-4fdf-b346-3f450d979bce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:46.912230 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-166290b3-6ce4-4c49-b665-a1639169f83d admin admin] PATH_INFO: `/v3/projects/6bafdf52ef5c4793becbab3a1b69cb08` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:46.913588 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-166290b3-6ce4-4c49-b665-a1639169f83d admin admin] RBAC: Authorizing `identity:delete_project(project_id=6bafdf52ef5c4793becbab3a1b69cb08)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:46.917070 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-166290b3-6ce4-4c49-b665-a1639169f83d admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:46.929999 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 911/1837] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:36:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 66 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:36:46.940581 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b037f52-73a5-4e61-87b6-7aa96d379d1d None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:46.941836 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b037f52-73a5-4e61-87b6-7aa96d379d1d None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:46.942499 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b037f52-73a5-4e61-87b6-7aa96d379d1d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:46.972313 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-0b037f52-73a5-4e61-87b6-7aa96d379d1d None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:36:46.973872 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-0b037f52-73a5-4e61-87b6-7aa96d379d1d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:46.992823 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-166290b3-6ce4-4c49-b665-a1639169f83d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6bafdf52ef5c4793becbab3a1b69cb08', 'request_id': 'req-166290b3-6ce4-4c49-b665-a1639169f83d'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:47.005329 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-166290b3-6ce4-4c49-b665-a1639169f83d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6bafdf52ef5c4793becbab3a1b69cb08', 'request_id': 'req-166290b3-6ce4-4c49-b665-a1639169f83d'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:47.008176 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=87,keystone:DELETE=11,keystone:INSERT=2 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:36:47.016746 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-166290b3-6ce4-4c49-b665-a1639169f83d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6bafdf52ef5c4793becbab3a1b69cb08', 'request_id': 'req-166290b3-6ce4-4c49-b665-a1639169f83d'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:47.037240 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-166290b3-6ce4-4c49-b665-a1639169f83d admin admin] The token cache is being invalidate because project 6bafdf52ef5c4793becbab3a1b69cb08 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81434) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:36:47.037730 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-166290b3-6ce4-4c49-b665-a1639169f83d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:47.038684 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-166290b3-6ce4-4c49-b665-a1639169f83d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:47.040669 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 927/1838] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:36:46 2022] DELETE /identity/v3/projects/6bafdf52ef5c4793becbab3a1b69cb08 => generated 0 bytes in 257 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 04:36:47.045460 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e19f0847-2606-4583-a9db-667d0e1b5e2a None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:47.046146 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e19f0847-2606-4583-a9db-667d0e1b5e2a None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:47.046456 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e19f0847-2606-4583-a9db-667d0e1b5e2a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:47.073128 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-e19f0847-2606-4583-a9db-667d0e1b5e2a None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:36:47.074359 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-e19f0847-2606-4583-a9db-667d0e1b5e2a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:47.075077 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 912/1839] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:36:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 04:36:47.080835 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b037f52-73a5-4e61-87b6-7aa96d379d1d None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:47.084997 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 928/1840] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:36:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:36:47.092809 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b037f52-73a5-4e61-87b6-7aa96d379d1d None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:47.095009 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3c5f086-8691-4d42-983c-03ab686d2a22 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:47.095680 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3c5f086-8691-4d42-983c-03ab686d2a22 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:47.095943 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3c5f086-8691-4d42-983c-03ab686d2a22 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:47.099488 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d858e386-44be-4753-88e5-d4213e01f1c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:47.100827 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d858e386-44be-4753-88e5-d4213e01f1c9 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:47.101339 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d858e386-44be-4753-88e5-d4213e01f1c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:47.101807 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d858e386-44be-4753-88e5-d4213e01f1c9 admin admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:47.102777 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d858e386-44be-4753-88e5-d4213e01f1c9 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:36:47.103331 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d858e386-44be-4753-88e5-d4213e01f1c9 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:47.105311 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d858e386-44be-4753-88e5-d4213e01f1c9 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:47.120988 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 913/1841] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Sep 25 04:36:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:36:47.125442 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d858e386-44be-4753-88e5-d4213e01f1c9 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:47.141460 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d858e386-44be-4753-88e5-d4213e01f1c9 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:47.150026 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-f3c5f086-8691-4d42-983c-03ab686d2a22 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:36:47.165623 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-f3c5f086-8691-4d42-983c-03ab686d2a22 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:47.172447 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8791d65-cefc-42fb-bfc2-f401f31b2da4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:47.174058 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8791d65-cefc-42fb-bfc2-f401f31b2da4 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:47.175278 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8791d65-cefc-42fb-bfc2-f401f31b2da4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:47.175278 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8791d65-cefc-42fb-bfc2-f401f31b2da4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:47.178082 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8791d65-cefc-42fb-bfc2-f401f31b2da4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:47.179365 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 929/1842] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:36:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 04:36:47.182763 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8791d65-cefc-42fb-bfc2-f401f31b2da4 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:47.185452 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3c5f086-8691-4d42-983c-03ab686d2a22 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:47.202619 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3c5f086-8691-4d42-983c-03ab686d2a22 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:47.216142 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0ab6484-dd75-4a4b-b0c0-0b78edca0846 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:47.216820 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0ab6484-dd75-4a4b-b0c0-0b78edca0846 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:47.217108 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0ab6484-dd75-4a4b-b0c0-0b78edca0846 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:47.217502 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0ab6484-dd75-4a4b-b0c0-0b78edca0846 admin admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:47.218175 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0ab6484-dd75-4a4b-b0c0-0b78edca0846 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:36:47.218486 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0ab6484-dd75-4a4b-b0c0-0b78edca0846 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:47.220115 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0ab6484-dd75-4a4b-b0c0-0b78edca0846 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:47.251184 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 914/1843] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:36:47 2022] POST /identity/v3/projects => generated 377 bytes in 126 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 04:36:47.252383 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8791d65-cefc-42fb-bfc2-f401f31b2da4 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:47.255568 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 930/1844] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Sep 25 04:36:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 70 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:36:47.259563 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0ab6484-dd75-4a4b-b0c0-0b78edca0846 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:47.271807 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0ab6484-dd75-4a4b-b0c0-0b78edca0846 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:47.388771 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2015b600-2b40-47d7-bad6-ad5413fcdf16 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:47.388771 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2015b600-2b40-47d7-bad6-ad5413fcdf16 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:47.388771 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2015b600-2b40-47d7-bad6-ad5413fcdf16 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:47.388771 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2015b600-2b40-47d7-bad6-ad5413fcdf16 admin admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:47.389184 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2015b600-2b40-47d7-bad6-ad5413fcdf16 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:47.390808 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2015b600-2b40-47d7-bad6-ad5413fcdf16 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:47.432173 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 931/1845] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:36:47 2022] POST /identity/v3/projects => generated 381 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 04:36:47.435943 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2015b600-2b40-47d7-bad6-ad5413fcdf16 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:47.441385 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8791d65-cefc-42fb-bfc2-f401f31b2da4 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:47.510670 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2015b600-2b40-47d7-bad6-ad5413fcdf16 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:47.516770 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdb852ff-9413-43d7-9039-3b776d1e20c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:47.517564 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdb852ff-9413-43d7-9039-3b776d1e20c0 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:47.517918 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdb852ff-9413-43d7-9039-3b776d1e20c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:47.518267 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdb852ff-9413-43d7-9039-3b776d1e20c0 admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:47.519024 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdb852ff-9413-43d7-9039-3b776d1e20c0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:47.521142 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdb852ff-9413-43d7-9039-3b776d1e20c0 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:47.522715 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:DELETE=2,keystone:SELECT=97,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:36:47.538035 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e118887e-2855-4d29-a22a-b9341b0fab32 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:47.538759 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e118887e-2855-4d29-a22a-b9341b0fab32 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:47.539158 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e118887e-2855-4d29-a22a-b9341b0fab32 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:47.540658 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e118887e-2855-4d29-a22a-b9341b0fab32 admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:47.541751 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e118887e-2855-4d29-a22a-b9341b0fab32 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:47.549838 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 932/1846] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:36:47 2022] POST /identity/v3/users => generated 291 bytes in 114 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:36:47.556149 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e118887e-2855-4d29-a22a-b9341b0fab32 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:47.558251 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdb852ff-9413-43d7-9039-3b776d1e20c0 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:47.588961 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdb852ff-9413-43d7-9039-3b776d1e20c0 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:47.588961 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dcc7891-dc64-4c88-8804-de671a5b7b13 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:47.588961 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dcc7891-dc64-4c88-8804-de671a5b7b13 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:47.588961 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dcc7891-dc64-4c88-8804-de671a5b7b13 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:47.589293 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dcc7891-dc64-4c88-8804-de671a5b7b13 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:47.589293 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dcc7891-dc64-4c88-8804-de671a5b7b13 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:47.592126 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dcc7891-dc64-4c88-8804-de671a5b7b13 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:47.605175 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 933/1847] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:36:47 2022] GET /identity/v3/roles => generated 1420 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:36:47.609429 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dcc7891-dc64-4c88-8804-de671a5b7b13 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:47.616351 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 915/1848] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:36:47 2022] POST /identity/v3/users => generated 289 bytes in 364 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:36:47.617647 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e118887e-2855-4d29-a22a-b9341b0fab32 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:47.622470 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dcc7891-dc64-4c88-8804-de671a5b7b13 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:47.627987 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41bd051e-675b-4050-9620-9e1433157f58 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:47.628733 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41bd051e-675b-4050-9620-9e1433157f58 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:47.629089 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41bd051e-675b-4050-9620-9e1433157f58 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:47.629477 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41bd051e-675b-4050-9620-9e1433157f58 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:47.630206 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41bd051e-675b-4050-9620-9e1433157f58 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:47.633304 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41bd051e-675b-4050-9620-9e1433157f58 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:47.640375 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e118887e-2855-4d29-a22a-b9341b0fab32 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:47.651645 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 934/1849] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:36:47 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:36:47.654054 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41bd051e-675b-4050-9620-9e1433157f58 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:47.655790 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bc19103-ce54-4fd3-b1af-1c8953d30414 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:47.656729 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bc19103-ce54-4fd3-b1af-1c8953d30414 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:47.658228 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bc19103-ce54-4fd3-b1af-1c8953d30414 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:47.658730 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bc19103-ce54-4fd3-b1af-1c8953d30414 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:47.659661 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bc19103-ce54-4fd3-b1af-1c8953d30414 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:47.662103 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bc19103-ce54-4fd3-b1af-1c8953d30414 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:47.666961 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41bd051e-675b-4050-9620-9e1433157f58 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:47.675596 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac83ab00-35c3-4e3e-8c55-ec55dc97d9cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:47.676802 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac83ab00-35c3-4e3e-8c55-ec55dc97d9cf admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:47.677497 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac83ab00-35c3-4e3e-8c55-ec55dc97d9cf admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:47.681997 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac83ab00-35c3-4e3e-8c55-ec55dc97d9cf admin admin] PATH_INFO: `/v3/projects/b1ccbe61726c46d7b7454882be6c8a0b/users/e17e9ce5fdc34359b434b6343a5f769a/roles/8f6cc574056c4278bc9778f14dc4b411` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:47.699247 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=95,keystone:DELETE=1,keystone:INSERT=4 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:36:47.702383 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac83ab00-35c3-4e3e-8c55-ec55dc97d9cf admin admin] RBAC: Authorizing `identity:create_grant(project_id=b1ccbe61726c46d7b7454882be6c8a0b, user_id=e17e9ce5fdc34359b434b6343a5f769a, role_id=8f6cc574056c4278bc9778f14dc4b411)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:47.706226 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 916/1850] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:36:47 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:36:47.706226 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bc19103-ce54-4fd3-b1af-1c8953d30414 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:47.706951 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-5bc19103-ce54-4fd3-b1af-1c8953d30414 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:47.710174 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac83ab00-35c3-4e3e-8c55-ec55dc97d9cf admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:47.725671 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bc19103-ce54-4fd3-b1af-1c8953d30414 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:47.733872 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 935/1851] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:36:47 2022] PUT /identity/v3/projects/b1ccbe61726c46d7b7454882be6c8a0b/users/e17e9ce5fdc34359b434b6343a5f769a/roles/8f6cc574056c4278bc9778f14dc4b411 => generated 0 bytes in 81 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:36:47.742607 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac83ab00-35c3-4e3e-8c55-ec55dc97d9cf admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:47.821898 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac83ab00-35c3-4e3e-8c55-ec55dc97d9cf admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:47.828186 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01ecddff-30d4-423e-81af-55b2af257b85 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:47.828516 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01ecddff-30d4-423e-81af-55b2af257b85 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:47.828781 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01ecddff-30d4-423e-81af-55b2af257b85 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:47.829067 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01ecddff-30d4-423e-81af-55b2af257b85 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:47.829724 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01ecddff-30d4-423e-81af-55b2af257b85 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:47.831775 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01ecddff-30d4-423e-81af-55b2af257b85 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:47.848279 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 936/1852] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:36:47 2022] GET /identity/v3/roles => generated 1420 bytes in 110 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:36:47.848279 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e70c592-d884-438a-892d-09648cde6ada None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:47.848279 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e70c592-d884-438a-892d-09648cde6ada None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:47.848279 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e70c592-d884-438a-892d-09648cde6ada None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:47.862958 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10ba0898-f650-4c90-9fde-9e383116e849 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:47.864662 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10ba0898-f650-4c90-9fde-9e383116e849 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:47.864972 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10ba0898-f650-4c90-9fde-9e383116e849 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:47.865256 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10ba0898-f650-4c90-9fde-9e383116e849 admin admin] PATH_INFO: `/v3/projects/26d0487282904512bffc6b4c356bcdd8` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:47.866777 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10ba0898-f650-4c90-9fde-9e383116e849 admin admin] RBAC: Authorizing `identity:delete_project(project_id=26d0487282904512bffc6b4c356bcdd8)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:47.868583 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10ba0898-f650-4c90-9fde-9e383116e849 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:47.891050 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-1e70c592-d884-438a-892d-09648cde6ada None None] MFA Rules not processed for user `e17e9ce5fdc34359b434b6343a5f769a`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:36:47.892296 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-1e70c592-d884-438a-892d-09648cde6ada None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:47.931144 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-10ba0898-f650-4c90-9fde-9e383116e849 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '26d0487282904512bffc6b4c356bcdd8', 'request_id': 'req-10ba0898-f650-4c90-9fde-9e383116e849'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:47.941253 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-10ba0898-f650-4c90-9fde-9e383116e849 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '26d0487282904512bffc6b4c356bcdd8', 'request_id': 'req-10ba0898-f650-4c90-9fde-9e383116e849'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:47.949765 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-10ba0898-f650-4c90-9fde-9e383116e849 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '26d0487282904512bffc6b4c356bcdd8', 'request_id': 'req-10ba0898-f650-4c90-9fde-9e383116e849'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:47.963742 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-10ba0898-f650-4c90-9fde-9e383116e849 admin admin] The token cache is being invalidate because project 26d0487282904512bffc6b4c356bcdd8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:36:47.964117 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-10ba0898-f650-4c90-9fde-9e383116e849 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:47.964835 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-10ba0898-f650-4c90-9fde-9e383116e849 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:47.967174 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 917/1853] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:36:47 2022] DELETE /identity/v3/projects/26d0487282904512bffc6b4c356bcdd8 => generated 0 bytes in 263 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:36:47.968043 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10ba0898-f650-4c90-9fde-9e383116e849 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:47.971075 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-10ba0898-f650-4c90-9fde-9e383116e849 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:47.977569 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 937/1854] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:36:47 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:36:47.978659 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e70c592-d884-438a-892d-09648cde6ada None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:47.991174 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e70c592-d884-438a-892d-09648cde6ada None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:48.053794 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10ba0898-f650-4c90-9fde-9e383116e849 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:48.058857 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e2f240e-e0c1-43c9-a3b0-be12d9f5dacc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:48.059651 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e2f240e-e0c1-43c9-a3b0-be12d9f5dacc admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:48.060564 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e2f240e-e0c1-43c9-a3b0-be12d9f5dacc admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:48.060564 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e2f240e-e0c1-43c9-a3b0-be12d9f5dacc admin admin] PATH_INFO: `/v3/projects/b6a625eeaecd4610906f007a4174fc0a/users/848b4af60f984dc788ae6994e89e4acf/roles/8f6cc574056c4278bc9778f14dc4b411` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:48.062972 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29238e2c-8406-485a-aebf-cf006804bf21 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:48.063422 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=89,keystone:INSERT=4,keystone:DELETE=7 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:36:48.063674 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29238e2c-8406-485a-aebf-cf006804bf21 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:48.063964 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29238e2c-8406-485a-aebf-cf006804bf21 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:48.064339 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29238e2c-8406-485a-aebf-cf006804bf21 admin admin] PATH_INFO: `/v3/users/95f823c6c5f844ddbfcd45b4afbb203e` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:48.065835 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29238e2c-8406-485a-aebf-cf006804bf21 admin admin] RBAC: Authorizing `identity:delete_user(user_id=95f823c6c5f844ddbfcd45b4afbb203e)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:48.068273 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29238e2c-8406-485a-aebf-cf006804bf21 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:48.087165 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e2f240e-e0c1-43c9-a3b0-be12d9f5dacc admin admin] RBAC: Authorizing `identity:create_grant(project_id=b6a625eeaecd4610906f007a4174fc0a, user_id=848b4af60f984dc788ae6994e89e4acf, role_id=8f6cc574056c4278bc9778f14dc4b411)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:48.090281 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=98,keystone:INSERT=1,keystone:DELETE=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:36:48.092009 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e2f240e-e0c1-43c9-a3b0-be12d9f5dacc admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:48.111440 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 918/1855] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:36:47 2022] PUT /identity/v3/projects/b6a625eeaecd4610906f007a4174fc0a/users/848b4af60f984dc788ae6994e89e4acf/roles/8f6cc574056c4278bc9778f14dc4b411 => generated 0 bytes in 144 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:36:48.112517 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e2f240e-e0c1-43c9-a3b0-be12d9f5dacc admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:48.123809 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-2e2f240e-e0c1-43c9-a3b0-be12d9f5dacc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:48.136465 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-29238e2c-8406-485a-aebf-cf006804bf21 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '95f823c6c5f844ddbfcd45b4afbb203e'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:48.147757 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-29238e2c-8406-485a-aebf-cf006804bf21 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '95f823c6c5f844ddbfcd45b4afbb203e'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:48.165345 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-29238e2c-8406-485a-aebf-cf006804bf21 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '95f823c6c5f844ddbfcd45b4afbb203e'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:48.166549 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-29238e2c-8406-485a-aebf-cf006804bf21 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '95f823c6c5f844ddbfcd45b4afbb203e'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:48.178912 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-29238e2c-8406-485a-aebf-cf006804bf21 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '95f823c6c5f844ddbfcd45b4afbb203e'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:48.185729 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 938/1856] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:36:47 2022] DELETE /identity/v3/users/95f823c6c5f844ddbfcd45b4afbb203e => generated 0 bytes in 207 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:36:48.186733 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84b1fbe4-8037-4b16-9cd2-5ba807e5514d None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:48.187024 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84b1fbe4-8037-4b16-9cd2-5ba807e5514d None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:48.187299 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84b1fbe4-8037-4b16-9cd2-5ba807e5514d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:48.201157 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e2f240e-e0c1-43c9-a3b0-be12d9f5dacc admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:48.226927 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-84b1fbe4-8037-4b16-9cd2-5ba807e5514d None None] MFA Rules not processed for user `848b4af60f984dc788ae6994e89e4acf`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:36:48.228066 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-84b1fbe4-8037-4b16-9cd2-5ba807e5514d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:48.265351 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bd8e10d-bf9d-4cd0-96f2-82b5e7fb12f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:48.266554 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bd8e10d-bf9d-4cd0-96f2-82b5e7fb12f1 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:48.266915 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bd8e10d-bf9d-4cd0-96f2-82b5e7fb12f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:48.267261 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bd8e10d-bf9d-4cd0-96f2-82b5e7fb12f1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:48.268455 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bd8e10d-bf9d-4cd0-96f2-82b5e7fb12f1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:48.270792 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bd8e10d-bf9d-4cd0-96f2-82b5e7fb12f1 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:48.311104 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 919/1857] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:36:48 2022] POST /identity/v3/projects => generated 383 bytes in 199 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:36:48.315766 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bd8e10d-bf9d-4cd0-96f2-82b5e7fb12f1 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:48.321668 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-6bd8e10d-bf9d-4cd0-96f2-82b5e7fb12f1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:48.338423 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 939/1858] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:36:48 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 152 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:36:48.344242 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84b1fbe4-8037-4b16-9cd2-5ba807e5514d None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:48.345683 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-84b1fbe4-8037-4b16-9cd2-5ba807e5514d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:48.361072 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84b1fbe4-8037-4b16-9cd2-5ba807e5514d None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:48.430977 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bd8e10d-bf9d-4cd0-96f2-82b5e7fb12f1 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:48.453397 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4951114-fced-45c1-9d12-a2c877209c0c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:48.454032 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4951114-fced-45c1-9d12-a2c877209c0c admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:48.454323 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4951114-fced-45c1-9d12-a2c877209c0c admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:48.454605 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4951114-fced-45c1-9d12-a2c877209c0c admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:48.455240 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4951114-fced-45c1-9d12-a2c877209c0c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:48.458502 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4951114-fced-45c1-9d12-a2c877209c0c admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:48.464970 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60b3e1cb-b05d-49e3-9f48-d6fd8ebe93c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:48.469820 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60b3e1cb-b05d-49e3-9f48-d6fd8ebe93c5 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:48.469820 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60b3e1cb-b05d-49e3-9f48-d6fd8ebe93c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:48.469820 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60b3e1cb-b05d-49e3-9f48-d6fd8ebe93c5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:48.469820 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60b3e1cb-b05d-49e3-9f48-d6fd8ebe93c5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:48.470430 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60b3e1cb-b05d-49e3-9f48-d6fd8ebe93c5 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:48.501646 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 920/1859] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:36:48 2022] POST /identity/v3/users => generated 298 bytes in 186 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:36:48.503055 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4951114-fced-45c1-9d12-a2c877209c0c admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:48.509640 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:36:48.511743 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 940/1860] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:36:48 2022] POST /identity/v3/projects => generated 375 bytes in 168 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:36:48.518069 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60b3e1cb-b05d-49e3-9f48-d6fd8ebe93c5 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:48.523668 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4951114-fced-45c1-9d12-a2c877209c0c admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:48.535927 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:DELETE=9,keystone:SELECT=89,keystone:INSERT=2 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:36:48.602551 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de5debd9-ec22-47f7-9b15-aad2a1b0d210 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:48.603332 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de5debd9-ec22-47f7-9b15-aad2a1b0d210 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:48.603628 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de5debd9-ec22-47f7-9b15-aad2a1b0d210 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:48.603909 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de5debd9-ec22-47f7-9b15-aad2a1b0d210 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:48.604669 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de5debd9-ec22-47f7-9b15-aad2a1b0d210 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:48.605952 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60b3e1cb-b05d-49e3-9f48-d6fd8ebe93c5 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:48.606388 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de5debd9-ec22-47f7-9b15-aad2a1b0d210 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:48.611265 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c91ab4d-ae9a-454e-b801-ba855c2ee9fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:48.612089 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c91ab4d-ae9a-454e-b801-ba855c2ee9fd admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:48.612908 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c91ab4d-ae9a-454e-b801-ba855c2ee9fd admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:48.612908 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c91ab4d-ae9a-454e-b801-ba855c2ee9fd admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:48.613802 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c91ab4d-ae9a-454e-b801-ba855c2ee9fd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:48.615815 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c91ab4d-ae9a-454e-b801-ba855c2ee9fd admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:48.620576 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 921/1861] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:36:48 2022] GET /identity/v3/roles => generated 1420 bytes in 118 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:36:48.625474 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de5debd9-ec22-47f7-9b15-aad2a1b0d210 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:48.652604 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de5debd9-ec22-47f7-9b15-aad2a1b0d210 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:48.658785 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b497f749-f5c7-4868-9f6f-6a7c920b595c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:48.659457 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b497f749-f5c7-4868-9f6f-6a7c920b595c admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:48.659741 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b497f749-f5c7-4868-9f6f-6a7c920b595c admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:48.660022 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b497f749-f5c7-4868-9f6f-6a7c920b595c admin admin] PATH_INFO: `/v3/projects/0996205223344abfa89c41bd1a033342/users/fe10488ddee84ae2a41e0340f266dbaf/roles/c51fe7c5ad804040879acfb54269455f` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:48.665750 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 941/1862] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:36:48 2022] POST /identity/v3/users => generated 294 bytes in 150 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 04:36:48.673462 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c91ab4d-ae9a-454e-b801-ba855c2ee9fd admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:48.681983 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b497f749-f5c7-4868-9f6f-6a7c920b595c admin admin] RBAC: Authorizing `identity:create_grant(project_id=0996205223344abfa89c41bd1a033342, user_id=fe10488ddee84ae2a41e0340f266dbaf, role_id=c51fe7c5ad804040879acfb54269455f)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:48.684013 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c91ab4d-ae9a-454e-b801-ba855c2ee9fd admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:48.685846 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b497f749-f5c7-4868-9f6f-6a7c920b595c admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:48.691365 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db9396bb-203a-4404-b4e2-89313ecab4b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:48.692157 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db9396bb-203a-4404-b4e2-89313ecab4b5 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:48.692929 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db9396bb-203a-4404-b4e2-89313ecab4b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:48.692929 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db9396bb-203a-4404-b4e2-89313ecab4b5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:48.693644 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db9396bb-203a-4404-b4e2-89313ecab4b5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:48.695521 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db9396bb-203a-4404-b4e2-89313ecab4b5 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:48.708044 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 922/1863] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:36:48 2022] PUT /identity/v3/projects/0996205223344abfa89c41bd1a033342/users/fe10488ddee84ae2a41e0340f266dbaf/roles/c51fe7c5ad804040879acfb54269455f => generated 0 bytes in 80 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 04:36:48.708044 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b497f749-f5c7-4868-9f6f-6a7c920b595c admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:48.708295 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-b497f749-f5c7-4868-9f6f-6a7c920b595c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:48.720865 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 942/1864] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:36:48 2022] GET /identity/v3/roles => generated 1420 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:36:48.721804 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db9396bb-203a-4404-b4e2-89313ecab4b5 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:48.736158 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db9396bb-203a-4404-b4e2-89313ecab4b5 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:48.780555 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b497f749-f5c7-4868-9f6f-6a7c920b595c admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:48.785872 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f5e2349-86a2-44e0-a90d-c3f119317282 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:48.786839 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f5e2349-86a2-44e0-a90d-c3f119317282 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:48.786975 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f5e2349-86a2-44e0-a90d-c3f119317282 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:48.787280 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f5e2349-86a2-44e0-a90d-c3f119317282 admin admin] PATH_INFO: `/v3/projects/26d4a30838dc41fb923f18b7aa73a3d9` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:48.788493 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f5e2349-86a2-44e0-a90d-c3f119317282 admin admin] RBAC: Authorizing `identity:delete_project(project_id=26d4a30838dc41fb923f18b7aa73a3d9)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:48.790516 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f5e2349-86a2-44e0-a90d-c3f119317282 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:48.801262 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9b94e41-0cef-44ea-8371-6f4e9f5fb2c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:48.801952 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9b94e41-0cef-44ea-8371-6f4e9f5fb2c5 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:48.802244 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9b94e41-0cef-44ea-8371-6f4e9f5fb2c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:48.802508 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9b94e41-0cef-44ea-8371-6f4e9f5fb2c5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:48.803201 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9b94e41-0cef-44ea-8371-6f4e9f5fb2c5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:48.804826 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9b94e41-0cef-44ea-8371-6f4e9f5fb2c5 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:48.816418 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 943/1865] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:36:48 2022] GET /identity/v3/roles => generated 1420 bytes in 95 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:36:48.817581 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9b94e41-0cef-44ea-8371-6f4e9f5fb2c5 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:48.830530 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9b94e41-0cef-44ea-8371-6f4e9f5fb2c5 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:48.846573 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95b452de-de9d-415b-8740-046e07cd19f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:48.847245 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95b452de-de9d-415b-8740-046e07cd19f1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:48.847525 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95b452de-de9d-415b-8740-046e07cd19f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:48.847801 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95b452de-de9d-415b-8740-046e07cd19f1 admin admin] PATH_INFO: `/v3/projects/b0ae2308c6424b6db559e7389154b104/users/30d27a152fd845e1806a39293f3a8ffe/roles/c51fe7c5ad804040879acfb54269455f` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:48.865405 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95b452de-de9d-415b-8740-046e07cd19f1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b0ae2308c6424b6db559e7389154b104, user_id=30d27a152fd845e1806a39293f3a8ffe, role_id=c51fe7c5ad804040879acfb54269455f)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:48.867533 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-6f5e2349-86a2-44e0-a90d-c3f119317282 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '26d4a30838dc41fb923f18b7aa73a3d9', 'request_id': 'req-6f5e2349-86a2-44e0-a90d-c3f119317282'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:48.868061 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95b452de-de9d-415b-8740-046e07cd19f1 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:48.877304 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-6f5e2349-86a2-44e0-a90d-c3f119317282 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '26d4a30838dc41fb923f18b7aa73a3d9', 'request_id': 'req-6f5e2349-86a2-44e0-a90d-c3f119317282'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:48.884999 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 944/1866] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:36:48 2022] PUT /identity/v3/projects/b0ae2308c6424b6db559e7389154b104/users/30d27a152fd845e1806a39293f3a8ffe/roles/c51fe7c5ad804040879acfb54269455f => generated 0 bytes in 68 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:36:48.886031 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95b452de-de9d-415b-8740-046e07cd19f1 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:48.890229 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-6f5e2349-86a2-44e0-a90d-c3f119317282 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '26d4a30838dc41fb923f18b7aa73a3d9', 'request_id': 'req-6f5e2349-86a2-44e0-a90d-c3f119317282'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:48.904339 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-6f5e2349-86a2-44e0-a90d-c3f119317282 admin admin] The token cache is being invalidate because project 26d4a30838dc41fb923f18b7aa73a3d9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:36:48.904782 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-6f5e2349-86a2-44e0-a90d-c3f119317282 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:48.905465 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-6f5e2349-86a2-44e0-a90d-c3f119317282 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:48.907389 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 923/1867] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:36:48 2022] DELETE /identity/v3/projects/26d4a30838dc41fb923f18b7aa73a3d9 => generated 0 bytes in 202 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:36:48.908503 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f5e2349-86a2-44e0-a90d-c3f119317282 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:48.909885 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-6f5e2349-86a2-44e0-a90d-c3f119317282 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:48.919676 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:36:48.958509 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95b452de-de9d-415b-8740-046e07cd19f1 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:48.963269 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e453216a-7f65-440c-b45c-24d903a18afc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:48.963936 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e453216a-7f65-440c-b45c-24d903a18afc admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:48.964215 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e453216a-7f65-440c-b45c-24d903a18afc admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:48.964492 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e453216a-7f65-440c-b45c-24d903a18afc admin admin] PATH_INFO: `/v3/projects/0996205223344abfa89c41bd1a033342/users/fe10488ddee84ae2a41e0340f266dbaf/roles/c51fe7c5ad804040879acfb54269455f` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:48.967353 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e453216a-7f65-440c-b45c-24d903a18afc admin admin] RBAC: Authorizing `identity:create_grant(project_id=0996205223344abfa89c41bd1a033342, user_id=fe10488ddee84ae2a41e0340f266dbaf, role_id=c51fe7c5ad804040879acfb54269455f)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:48.970065 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e453216a-7f65-440c-b45c-24d903a18afc admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:48.972573 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:36:48.984883 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 945/1868] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:36:48 2022] PUT /identity/v3/projects/0996205223344abfa89c41bd1a033342/users/fe10488ddee84ae2a41e0340f266dbaf/roles/c51fe7c5ad804040879acfb54269455f => generated 0 bytes in 99 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:36:48.986233 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89c71088-28d0-452f-8a56-9bda9cb90ab0 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:48.986722 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89c71088-28d0-452f-8a56-9bda9cb90ab0 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:48.993023 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89c71088-28d0-452f-8a56-9bda9cb90ab0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:48.993440 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f5e2349-86a2-44e0-a90d-c3f119317282 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:49.016859 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-89c71088-28d0-452f-8a56-9bda9cb90ab0 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:36:49.018115 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-89c71088-28d0-452f-8a56-9bda9cb90ab0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:49.058841 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8686b9c7-7853-44d5-a23d-993bbcae4edd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:49.060620 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8686b9c7-7853-44d5-a23d-993bbcae4edd admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:49.060934 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8686b9c7-7853-44d5-a23d-993bbcae4edd admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:49.061208 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8686b9c7-7853-44d5-a23d-993bbcae4edd admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:49.061858 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8686b9c7-7853-44d5-a23d-993bbcae4edd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:49.063638 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8686b9c7-7853-44d5-a23d-993bbcae4edd admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:49.073694 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 924/1869] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:36:48 2022] GET /identity/v3/roles => generated 1420 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:36:49.073694 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8185bab7-f33d-48ae-9cec-7243c003a1bb None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:49.073694 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8185bab7-f33d-48ae-9cec-7243c003a1bb None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:49.073694 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8185bab7-f33d-48ae-9cec-7243c003a1bb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:49.081506 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 946/1870] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:36:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:36:49.082645 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89c71088-28d0-452f-8a56-9bda9cb90ab0 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:49.093020 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89c71088-28d0-452f-8a56-9bda9cb90ab0 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:49.097843 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-897fac50-a562-4755-9f91-8d72c355b6d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:49.098538 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-897fac50-a562-4755-9f91-8d72c355b6d3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:49.098848 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-897fac50-a562-4755-9f91-8d72c355b6d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:49.099115 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-897fac50-a562-4755-9f91-8d72c355b6d3 admin admin] PATH_INFO: `/v3/projects/b0ae2308c6424b6db559e7389154b104/users/30d27a152fd845e1806a39293f3a8ffe/roles/c51fe7c5ad804040879acfb54269455f` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:49.101111 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-897fac50-a562-4755-9f91-8d72c355b6d3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b0ae2308c6424b6db559e7389154b104, user_id=30d27a152fd845e1806a39293f3a8ffe, role_id=c51fe7c5ad804040879acfb54269455f)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:49.103774 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-897fac50-a562-4755-9f91-8d72c355b6d3 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:49.109827 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-8185bab7-f33d-48ae-9cec-7243c003a1bb None None] MFA Rules not processed for user `fe10488ddee84ae2a41e0340f266dbaf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:36:49.111046 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-8185bab7-f33d-48ae-9cec-7243c003a1bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:49.115410 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 947/1871] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:36:49 2022] PUT /identity/v3/projects/b0ae2308c6424b6db559e7389154b104/users/30d27a152fd845e1806a39293f3a8ffe/roles/c51fe7c5ad804040879acfb54269455f => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:36:49.122002 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42a1b330-b83b-4957-90af-bcab4f8dbeae None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:49.122277 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42a1b330-b83b-4957-90af-bcab4f8dbeae None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:49.122558 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42a1b330-b83b-4957-90af-bcab4f8dbeae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:49.142678 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-42a1b330-b83b-4957-90af-bcab4f8dbeae None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:36:49.143880 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-42a1b330-b83b-4957-90af-bcab4f8dbeae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:49.180466 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 925/1872] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:36:49 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:36:49.181910 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f62f674-537a-4d90-8a91-13713d37bfa0 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:49.182199 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f62f674-537a-4d90-8a91-13713d37bfa0 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:49.182476 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f62f674-537a-4d90-8a91-13713d37bfa0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:49.208433 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 948/1873] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:36:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:36:49.209306 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42a1b330-b83b-4957-90af-bcab4f8dbeae None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:49.210644 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-42a1b330-b83b-4957-90af-bcab4f8dbeae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:49.214295 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-9f62f674-537a-4d90-8a91-13713d37bfa0 None None] MFA Rules not processed for user `30d27a152fd845e1806a39293f3a8ffe`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:36:49.215484 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-9f62f674-537a-4d90-8a91-13713d37bfa0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:49.240000 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:36:49.275834 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42a1b330-b83b-4957-90af-bcab4f8dbeae None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:49.280203 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1d644da-12e6-4581-89b5-829524d856ac service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ffbc136c77b84e8c87128f7247f4ec03', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:49.280981 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1d644da-12e6-4581-89b5-829524d856ac service cinder] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:49.281235 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1d644da-12e6-4581-89b5-829524d856ac service cinder] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:49.281424 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1d644da-12e6-4581-89b5-829524d856ac service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:49.291123 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1d644da-12e6-4581-89b5-829524d856ac service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:49.291405 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 926/1874] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:36:49 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:36:49.292278 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f62f674-537a-4d90-8a91-13713d37bfa0 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:49.292802 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1d644da-12e6-4581-89b5-829524d856ac service cinder] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:49.303168 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f62f674-537a-4d90-8a91-13713d37bfa0 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:49.304747 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 949/1875] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:36:49 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:36:49.305583 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1d644da-12e6-4581-89b5-829524d856ac service cinder] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:49.306932 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-a1d644da-12e6-4581-89b5-829524d856ac service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:49.308662 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d228607b-745b-493e-9c49-0d056b0041b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:49.310146 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d228607b-745b-493e-9c49-0d056b0041b8 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:49.310582 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d228607b-745b-493e-9c49-0d056b0041b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:49.311007 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d228607b-745b-493e-9c49-0d056b0041b8 admin admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:49.311863 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d228607b-745b-493e-9c49-0d056b0041b8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:36:49.312319 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d228607b-745b-493e-9c49-0d056b0041b8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:49.314123 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d228607b-745b-493e-9c49-0d056b0041b8 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:49.320138 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1d644da-12e6-4581-89b5-829524d856ac service cinder] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:49.324135 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da0dee7d-03e4-4fdc-9c2e-8490a20fc80a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ffbc136c77b84e8c87128f7247f4ec03', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:49.325824 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da0dee7d-03e4-4fdc-9c2e-8490a20fc80a service cinder] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:49.326144 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da0dee7d-03e4-4fdc-9c2e-8490a20fc80a service cinder] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:49.326429 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da0dee7d-03e4-4fdc-9c2e-8490a20fc80a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:49.328860 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-da0dee7d-03e4-4fdc-9c2e-8490a20fc80a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:49.331318 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 927/1876] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Sep 25 04:36:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:36:49.335866 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d228607b-745b-493e-9c49-0d056b0041b8 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:49.348245 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d228607b-745b-493e-9c49-0d056b0041b8 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:49.353221 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9f4b4ee-72f4-45f8-8dc1-db3f63db8a2c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:49.354167 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9f4b4ee-72f4-45f8-8dc1-db3f63db8a2c admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:49.355177 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9f4b4ee-72f4-45f8-8dc1-db3f63db8a2c admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:49.356210 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9f4b4ee-72f4-45f8-8dc1-db3f63db8a2c admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:49.357493 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9f4b4ee-72f4-45f8-8dc1-db3f63db8a2c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:49.361424 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9f4b4ee-72f4-45f8-8dc1-db3f63db8a2c admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:49.364314 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:INSERT=2,keystone:SELECT=98 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:36:49.410237 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 928/1877] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:36:49 2022] POST /identity/v3/projects => generated 387 bytes in 75 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:36:49.411541 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9f4b4ee-72f4-45f8-8dc1-db3f63db8a2c admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:49.422993 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da0dee7d-03e4-4fdc-9c2e-8490a20fc80a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:49.424467 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da0dee7d-03e4-4fdc-9c2e-8490a20fc80a service cinder] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:49.434672 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 950/1878] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:36:49 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 129 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:36:49.435358 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da0dee7d-03e4-4fdc-9c2e-8490a20fc80a service cinder] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:49.446482 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da0dee7d-03e4-4fdc-9c2e-8490a20fc80a service cinder] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:49.488372 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9f4b4ee-72f4-45f8-8dc1-db3f63db8a2c admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:49.492884 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01031045-a3bd-4067-810b-737e53ccec90 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ffbc136c77b84e8c87128f7247f4ec03', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:49.494928 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01031045-a3bd-4067-810b-737e53ccec90 service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:49.495213 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01031045-a3bd-4067-810b-737e53ccec90 service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:49.495485 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01031045-a3bd-4067-810b-737e53ccec90 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:49.498760 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-01031045-a3bd-4067-810b-737e53ccec90 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:49.519133 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8234d14d-904e-482a-89a9-42cc07485829 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:49.519784 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8234d14d-904e-482a-89a9-42cc07485829 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:49.520066 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8234d14d-904e-482a-89a9-42cc07485829 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:49.520337 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8234d14d-904e-482a-89a9-42cc07485829 admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:49.521013 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8234d14d-904e-482a-89a9-42cc07485829 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:49.523095 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8234d14d-904e-482a-89a9-42cc07485829 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:49.552706 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 951/1879] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:36:49 2022] POST /identity/v3/users => generated 294 bytes in 118 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:36:49.556953 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8234d14d-904e-482a-89a9-42cc07485829 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:49.567819 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8234d14d-904e-482a-89a9-42cc07485829 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:49.573202 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e732ea34-d078-4cf1-a7c3-568d772755e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:49.573406 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01031045-a3bd-4067-810b-737e53ccec90 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:49.573813 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e732ea34-d078-4cf1-a7c3-568d772755e8 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:49.574127 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e732ea34-d078-4cf1-a7c3-568d772755e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:49.574376 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e732ea34-d078-4cf1-a7c3-568d772755e8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:49.574918 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01031045-a3bd-4067-810b-737e53ccec90 service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:49.575110 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e732ea34-d078-4cf1-a7c3-568d772755e8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:49.576863 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e732ea34-d078-4cf1-a7c3-568d772755e8 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:49.587289 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 929/1880] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:36:49 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 176 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:36:49.587798 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 952/1881] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:36:49 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 25 04:36:49.592625 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01031045-a3bd-4067-810b-737e53ccec90 service cinder] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:49.606614 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01031045-a3bd-4067-810b-737e53ccec90 service cinder] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:49.612152 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-873262c7-de32-4455-b1a7-b68d59275c15 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:49.617805 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-873262c7-de32-4455-b1a7-b68d59275c15 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:49.618084 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-873262c7-de32-4455-b1a7-b68d59275c15 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:49.618441 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-873262c7-de32-4455-b1a7-b68d59275c15 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:49.619226 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-873262c7-de32-4455-b1a7-b68d59275c15 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:49.621082 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-873262c7-de32-4455-b1a7-b68d59275c15 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:49.636838 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 930/1882] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:36:49 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:36:49.638872 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e732ea34-d078-4cf1-a7c3-568d772755e8 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:49.660240 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e732ea34-d078-4cf1-a7c3-568d772755e8 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:49.666122 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-460f14f6-fa6e-415c-a20e-36d46baee8b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:49.668514 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-460f14f6-fa6e-415c-a20e-36d46baee8b8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:49.669599 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-460f14f6-fa6e-415c-a20e-36d46baee8b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:49.669884 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-460f14f6-fa6e-415c-a20e-36d46baee8b8 admin admin] PATH_INFO: `/v3/projects/bdfa54b3dbf4404f8c32c953d2b49e72/users/74a14b3a3c884f6b9177ac5f66c65e82/roles/8f6cc574056c4278bc9778f14dc4b411` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:49.688444 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-460f14f6-fa6e-415c-a20e-36d46baee8b8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bdfa54b3dbf4404f8c32c953d2b49e72, user_id=74a14b3a3c884f6b9177ac5f66c65e82, role_id=8f6cc574056c4278bc9778f14dc4b411)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:49.692738 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-460f14f6-fa6e-415c-a20e-36d46baee8b8 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:49.724106 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 953/1883] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:36:49 2022] PUT /identity/v3/projects/bdfa54b3dbf4404f8c32c953d2b49e72/users/74a14b3a3c884f6b9177ac5f66c65e82/roles/8f6cc574056c4278bc9778f14dc4b411 => generated 0 bytes in 76 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:36:49.728832 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0436446e-bfcb-4d91-b2bf-be53c415cb67 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:49.729231 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0436446e-bfcb-4d91-b2bf-be53c415cb67 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:49.729717 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0436446e-bfcb-4d91-b2bf-be53c415cb67 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:49.772620 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-0436446e-bfcb-4d91-b2bf-be53c415cb67 None None] MFA Rules not processed for user `74a14b3a3c884f6b9177ac5f66c65e82`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:36:49.774943 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-0436446e-bfcb-4d91-b2bf-be53c415cb67 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:49.799317 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:36:49.884795 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 931/1884] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:36:49 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 159 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:36:49.889807 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-460f14f6-fa6e-415c-a20e-36d46baee8b8 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:49.970629 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-460f14f6-fa6e-415c-a20e-36d46baee8b8 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:49.976386 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f321a98e-da0b-4679-8999-24ee2cd57838 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:49.977244 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f321a98e-da0b-4679-8999-24ee2cd57838 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:49.977533 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f321a98e-da0b-4679-8999-24ee2cd57838 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:49.980755 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f321a98e-da0b-4679-8999-24ee2cd57838 admin admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:49.980755 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f321a98e-da0b-4679-8999-24ee2cd57838 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:49.981146 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f321a98e-da0b-4679-8999-24ee2cd57838 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:49.982269 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0436446e-bfcb-4d91-b2bf-be53c415cb67 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:49.983912 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-0436446e-bfcb-4d91-b2bf-be53c415cb67 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:49.998896 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0436446e-bfcb-4d91-b2bf-be53c415cb67 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:50.004282 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d01a08a4-b342-4e27-b085-d0b3cdb0f282 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:50.005016 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d01a08a4-b342-4e27-b085-d0b3cdb0f282 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:50.005507 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d01a08a4-b342-4e27-b085-d0b3cdb0f282 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:50.006012 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d01a08a4-b342-4e27-b085-d0b3cdb0f282 admin admin] PATH_INFO: `/v3/users/e17e9ce5fdc34359b434b6343a5f769a` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:50.007302 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d01a08a4-b342-4e27-b085-d0b3cdb0f282 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e17e9ce5fdc34359b434b6343a5f769a)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:50.009437 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d01a08a4-b342-4e27-b085-d0b3cdb0f282 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:50.020787 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 954/1885] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:36:49 2022] POST /identity/v3/projects => generated 387 bytes in 131 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:36:50.026544 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f321a98e-da0b-4679-8999-24ee2cd57838 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:50.031720 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:36:50.068543 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-d01a08a4-b342-4e27-b085-d0b3cdb0f282 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e17e9ce5fdc34359b434b6343a5f769a'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:50.078821 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-d01a08a4-b342-4e27-b085-d0b3cdb0f282 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e17e9ce5fdc34359b434b6343a5f769a'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:50.092591 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-d01a08a4-b342-4e27-b085-d0b3cdb0f282 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e17e9ce5fdc34359b434b6343a5f769a'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:50.093482 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-d01a08a4-b342-4e27-b085-d0b3cdb0f282 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e17e9ce5fdc34359b434b6343a5f769a'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:50.101507 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-d01a08a4-b342-4e27-b085-d0b3cdb0f282 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e17e9ce5fdc34359b434b6343a5f769a'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:50.106874 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f321a98e-da0b-4679-8999-24ee2cd57838 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:50.109480 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 932/1886] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:36:49 2022] DELETE /identity/v3/users/e17e9ce5fdc34359b434b6343a5f769a => generated 0 bytes in 128 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 04:36:50.117466 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-804b0126-c428-40fd-81b6-770c1c131b08 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:50.118288 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-804b0126-c428-40fd-81b6-770c1c131b08 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:50.118575 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-804b0126-c428-40fd-81b6-770c1c131b08 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:50.118851 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-804b0126-c428-40fd-81b6-770c1c131b08 admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:50.119488 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-804b0126-c428-40fd-81b6-770c1c131b08 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:50.121597 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-804b0126-c428-40fd-81b6-770c1c131b08 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:50.126097 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f1ecd58-83d1-47af-90ec-2e95cb18764a None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:50.126502 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f1ecd58-83d1-47af-90ec-2e95cb18764a None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:50.126882 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f1ecd58-83d1-47af-90ec-2e95cb18764a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:50.148257 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-2f1ecd58-83d1-47af-90ec-2e95cb18764a None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:36:50.149669 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-2f1ecd58-83d1-47af-90ec-2e95cb18764a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:50.150741 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 955/1887] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:36:50 2022] POST /identity/v3/users => generated 300 bytes in 125 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:36:50.154703 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-804b0126-c428-40fd-81b6-770c1c131b08 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:50.156423 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-804b0126-c428-40fd-81b6-770c1c131b08 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:50.186600 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-804b0126-c428-40fd-81b6-770c1c131b08 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:50.232092 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 933/1888] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:36:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:36:50.245054 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f1ecd58-83d1-47af-90ec-2e95cb18764a None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:50.248521 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-2f1ecd58-83d1-47af-90ec-2e95cb18764a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:50.284581 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33420306-cf7d-4e3e-8562-3df11162380a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:50.285464 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33420306-cf7d-4e3e-8562-3df11162380a admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:50.285931 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33420306-cf7d-4e3e-8562-3df11162380a admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:50.286406 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33420306-cf7d-4e3e-8562-3df11162380a admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:50.287453 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33420306-cf7d-4e3e-8562-3df11162380a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:50.289567 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33420306-cf7d-4e3e-8562-3df11162380a admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:50.306235 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 956/1889] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:36:50 2022] GET /identity/v3/roles => generated 1420 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 25 04:36:50.311078 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33420306-cf7d-4e3e-8562-3df11162380a admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:50.323349 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33420306-cf7d-4e3e-8562-3df11162380a admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:50.329631 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-950df650-c789-43de-9053-fe6b23e49525 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:50.330603 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-950df650-c789-43de-9053-fe6b23e49525 admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:50.331329 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-950df650-c789-43de-9053-fe6b23e49525 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:50.331784 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-950df650-c789-43de-9053-fe6b23e49525 admin admin] PATH_INFO: `/v3/projects/740000af86e94b97b973dd68f26c4d2d/users/b3ca94b4e4ec43a6892b52af62b0e477/roles/c51fe7c5ad804040879acfb54269455f` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:50.333449 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:36:50.351474 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-950df650-c789-43de-9053-fe6b23e49525 admin admin] RBAC: Authorizing `identity:create_grant(project_id=740000af86e94b97b973dd68f26c4d2d, user_id=b3ca94b4e4ec43a6892b52af62b0e477, role_id=c51fe7c5ad804040879acfb54269455f)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:50.354648 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-950df650-c789-43de-9053-fe6b23e49525 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:50.366443 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f1ecd58-83d1-47af-90ec-2e95cb18764a None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:50.370885 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 957/1890] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:36:50 2022] PUT /identity/v3/projects/740000af86e94b97b973dd68f26c4d2d/users/b3ca94b4e4ec43a6892b52af62b0e477/roles/c51fe7c5ad804040879acfb54269455f => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 04:36:50.375226 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-950df650-c789-43de-9053-fe6b23e49525 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:50.427602 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-695b7895-b355-45bd-ba2b-9dfcf8a913d4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:50.428247 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-695b7895-b355-45bd-ba2b-9dfcf8a913d4 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:50.428528 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-695b7895-b355-45bd-ba2b-9dfcf8a913d4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:50.428829 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-695b7895-b355-45bd-ba2b-9dfcf8a913d4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:50.444875 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-950df650-c789-43de-9053-fe6b23e49525 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:50.450410 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dea80f3-5d1b-4070-928d-bf22eefd83e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:50.451662 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dea80f3-5d1b-4070-928d-bf22eefd83e6 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:50.452027 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dea80f3-5d1b-4070-928d-bf22eefd83e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:50.452366 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dea80f3-5d1b-4070-928d-bf22eefd83e6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:50.453180 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dea80f3-5d1b-4070-928d-bf22eefd83e6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:50.454910 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dea80f3-5d1b-4070-928d-bf22eefd83e6 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:50.462217 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:36:50.475085 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 958/1891] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:36:50 2022] GET /identity/v3/roles => generated 1420 bytes in 98 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 25 04:36:50.482030 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dea80f3-5d1b-4070-928d-bf22eefd83e6 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:50.494335 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dea80f3-5d1b-4070-928d-bf22eefd83e6 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:50.499623 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-695b7895-b355-45bd-ba2b-9dfcf8a913d4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:50.501196 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-695b7895-b355-45bd-ba2b-9dfcf8a913d4 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:50.501688 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27d2ffd8-3576-4a03-b1a3-977e0c090546 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:50.502362 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27d2ffd8-3576-4a03-b1a3-977e0c090546 admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:50.502712 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27d2ffd8-3576-4a03-b1a3-977e0c090546 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:50.502931 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27d2ffd8-3576-4a03-b1a3-977e0c090546 admin admin] PATH_INFO: `/v3/projects/740000af86e94b97b973dd68f26c4d2d/users/b3ca94b4e4ec43a6892b52af62b0e477/roles/c51fe7c5ad804040879acfb54269455f` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:50.504938 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27d2ffd8-3576-4a03-b1a3-977e0c090546 admin admin] RBAC: Authorizing `identity:create_grant(project_id=740000af86e94b97b973dd68f26c4d2d, user_id=b3ca94b4e4ec43a6892b52af62b0e477, role_id=c51fe7c5ad804040879acfb54269455f)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:50.508448 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27d2ffd8-3576-4a03-b1a3-977e0c090546 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:50.511950 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 934/1892] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:36:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 267 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:36:50.522938 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 959/1893] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:36:50 2022] PUT /identity/v3/projects/740000af86e94b97b973dd68f26c4d2d/users/b3ca94b4e4ec43a6892b52af62b0e477/roles/c51fe7c5ad804040879acfb54269455f => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:36:50.533970 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-418ed17c-5783-4893-ab3e-876132333799 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:50.534269 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-418ed17c-5783-4893-ab3e-876132333799 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:50.534551 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-418ed17c-5783-4893-ab3e-876132333799 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:50.567366 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-418ed17c-5783-4893-ab3e-876132333799 None None] MFA Rules not processed for user `b3ca94b4e4ec43a6892b52af62b0e477`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:36:50.568760 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-418ed17c-5783-4893-ab3e-876132333799 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:50.666338 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 935/1894] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:36:50 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:36:50.679256 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27d2ffd8-3576-4a03-b1a3-977e0c090546 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:50.681279 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-27d2ffd8-3576-4a03-b1a3-977e0c090546 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:50.743606 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27d2ffd8-3576-4a03-b1a3-977e0c090546 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:50.747636 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03a136a8-0bda-4246-8382-0e4ff36407df service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ffbc136c77b84e8c87128f7247f4ec03', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:50.748321 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03a136a8-0bda-4246-8382-0e4ff36407df service cinder] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:50.748717 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03a136a8-0bda-4246-8382-0e4ff36407df service cinder] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:50.749064 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03a136a8-0bda-4246-8382-0e4ff36407df service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:50.758877 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03a136a8-0bda-4246-8382-0e4ff36407df service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:50.760323 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03a136a8-0bda-4246-8382-0e4ff36407df service cinder] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:50.770937 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 960/1895] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:36:50 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:36:50.803604 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-418ed17c-5783-4893-ab3e-876132333799 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:50.809319 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03a136a8-0bda-4246-8382-0e4ff36407df service cinder] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:50.811193 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-03a136a8-0bda-4246-8382-0e4ff36407df service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:50.835573 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:36:50.884804 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-418ed17c-5783-4893-ab3e-876132333799 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:50.889836 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a539a306-2c05-4083-adb5-40dc603582e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:50.890479 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a539a306-2c05-4083-adb5-40dc603582e4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:50.890775 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a539a306-2c05-4083-adb5-40dc603582e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:50.891030 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a539a306-2c05-4083-adb5-40dc603582e4 admin admin] PATH_INFO: `/v3/users/74a14b3a3c884f6b9177ac5f66c65e82` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:50.892181 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a539a306-2c05-4083-adb5-40dc603582e4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=74a14b3a3c884f6b9177ac5f66c65e82)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:50.894139 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a539a306-2c05-4083-adb5-40dc603582e4 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:50.899971 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03a136a8-0bda-4246-8382-0e4ff36407df service cinder] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:50.912488 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc6b65f9-52a4-449c-94e8-768847a5aa0c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:50.913360 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc6b65f9-52a4-449c-94e8-768847a5aa0c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:50.913714 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc6b65f9-52a4-449c-94e8-768847a5aa0c admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:50.914059 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc6b65f9-52a4-449c-94e8-768847a5aa0c admin admin] PATH_INFO: `/v3/projects/b1ccbe61726c46d7b7454882be6c8a0b` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:50.915282 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc6b65f9-52a4-449c-94e8-768847a5aa0c admin admin] RBAC: Authorizing `identity:delete_project(project_id=b1ccbe61726c46d7b7454882be6c8a0b)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:50.920282 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc6b65f9-52a4-449c-94e8-768847a5aa0c admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:50.947665 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-a539a306-2c05-4083-adb5-40dc603582e4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '74a14b3a3c884f6b9177ac5f66c65e82'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:50.959321 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-a539a306-2c05-4083-adb5-40dc603582e4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '74a14b3a3c884f6b9177ac5f66c65e82'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:50.976089 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-a539a306-2c05-4083-adb5-40dc603582e4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '74a14b3a3c884f6b9177ac5f66c65e82'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:50.977022 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-a539a306-2c05-4083-adb5-40dc603582e4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '74a14b3a3c884f6b9177ac5f66c65e82'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:50.987245 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-a539a306-2c05-4083-adb5-40dc603582e4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '74a14b3a3c884f6b9177ac5f66c65e82'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:50.994520 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-dc6b65f9-52a4-449c-94e8-768847a5aa0c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b1ccbe61726c46d7b7454882be6c8a0b', 'request_id': 'req-dc6b65f9-52a4-449c-94e8-768847a5aa0c'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:50.996371 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 936/1896] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:36:50 2022] DELETE /identity/v3/users/74a14b3a3c884f6b9177ac5f66c65e82 => generated 0 bytes in 193 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:36:51.005486 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f4d4516-7693-45c5-ba26-da024974ca30 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:51.007178 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-dc6b65f9-52a4-449c-94e8-768847a5aa0c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b1ccbe61726c46d7b7454882be6c8a0b', 'request_id': 'req-dc6b65f9-52a4-449c-94e8-768847a5aa0c'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:51.007753 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f4d4516-7693-45c5-ba26-da024974ca30 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:51.008454 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f4d4516-7693-45c5-ba26-da024974ca30 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:51.015356 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-dc6b65f9-52a4-449c-94e8-768847a5aa0c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b1ccbe61726c46d7b7454882be6c8a0b', 'request_id': 'req-dc6b65f9-52a4-449c-94e8-768847a5aa0c'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:51.034204 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-dc6b65f9-52a4-449c-94e8-768847a5aa0c admin admin] The token cache is being invalidate because project b1ccbe61726c46d7b7454882be6c8a0b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81434) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:36:51.034673 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-dc6b65f9-52a4-449c-94e8-768847a5aa0c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:51.035474 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-dc6b65f9-52a4-449c-94e8-768847a5aa0c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:51.040100 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 961/1897] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:36:50 2022] DELETE /identity/v3/projects/b1ccbe61726c46d7b7454882be6c8a0b => generated 0 bytes in 229 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 04:36:51.041223 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc6b65f9-52a4-449c-94e8-768847a5aa0c admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:51.042064 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-8f4d4516-7693-45c5-ba26-da024974ca30 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:36:51.042818 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-dc6b65f9-52a4-449c-94e8-768847a5aa0c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:51.043469 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-8f4d4516-7693-45c5-ba26-da024974ca30 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:51.081038 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:36:51.145802 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 937/1898] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:36:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:36:51.160111 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f4d4516-7693-45c5-ba26-da024974ca30 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:51.162714 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-8f4d4516-7693-45c5-ba26-da024974ca30 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:51.179079 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc6b65f9-52a4-449c-94e8-768847a5aa0c admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:51.187268 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b3eb0ed-5220-4107-bc7e-f4d2f20614e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:51.188076 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b3eb0ed-5220-4107-bc7e-f4d2f20614e8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:51.188905 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b3eb0ed-5220-4107-bc7e-f4d2f20614e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:51.189261 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b3eb0ed-5220-4107-bc7e-f4d2f20614e8 admin admin] PATH_INFO: `/v3/users/fe10488ddee84ae2a41e0340f266dbaf` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:51.190792 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b3eb0ed-5220-4107-bc7e-f4d2f20614e8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=fe10488ddee84ae2a41e0340f266dbaf)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:51.193914 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b3eb0ed-5220-4107-bc7e-f4d2f20614e8 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:51.240833 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:36:51.274410 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f4d4516-7693-45c5-ba26-da024974ca30 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:51.279758 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8355e97e-a023-49ea-86f4-80088f1a841c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:51.280607 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8355e97e-a023-49ea-86f4-80088f1a841c service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:51.281873 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8355e97e-a023-49ea-86f4-80088f1a841c service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:51.281873 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8355e97e-a023-49ea-86f4-80088f1a841c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:51.288314 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-4b3eb0ed-5220-4107-bc7e-f4d2f20614e8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fe10488ddee84ae2a41e0340f266dbaf'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:51.297062 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8355e97e-a023-49ea-86f4-80088f1a841c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:51.298874 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8355e97e-a023-49ea-86f4-80088f1a841c service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:51.306580 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-4b3eb0ed-5220-4107-bc7e-f4d2f20614e8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fe10488ddee84ae2a41e0340f266dbaf'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:51.311024 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 938/1899] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:36:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 152 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 25 04:36:51.322230 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-4b3eb0ed-5220-4107-bc7e-f4d2f20614e8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fe10488ddee84ae2a41e0340f266dbaf'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:51.322970 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-4b3eb0ed-5220-4107-bc7e-f4d2f20614e8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fe10488ddee84ae2a41e0340f266dbaf'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:51.331320 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-4b3eb0ed-5220-4107-bc7e-f4d2f20614e8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fe10488ddee84ae2a41e0340f266dbaf'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:51.337959 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 962/1900] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:36:51 2022] DELETE /identity/v3/users/fe10488ddee84ae2a41e0340f266dbaf => generated 0 bytes in 297 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 04:36:51.656493 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8355e97e-a023-49ea-86f4-80088f1a841c service neutron] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:51.670754 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8355e97e-a023-49ea-86f4-80088f1a841c service neutron] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:51.770172 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f278a78-f808-4cf1-a2e6-c27cc8a2f048 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:51.770842 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f278a78-f808-4cf1-a2e6-c27cc8a2f048 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:51.771217 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f278a78-f808-4cf1-a2e6-c27cc8a2f048 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:51.771404 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f278a78-f808-4cf1-a2e6-c27cc8a2f048 admin admin] PATH_INFO: `/v3/projects/0996205223344abfa89c41bd1a033342` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:51.772708 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f278a78-f808-4cf1-a2e6-c27cc8a2f048 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0996205223344abfa89c41bd1a033342)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:51.774950 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f278a78-f808-4cf1-a2e6-c27cc8a2f048 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:51.794327 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b3eb0ed-5220-4107-bc7e-f4d2f20614e8 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:51.797254 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-4b3eb0ed-5220-4107-bc7e-f4d2f20614e8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:51.825707 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b3eb0ed-5220-4107-bc7e-f4d2f20614e8 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:51.833014 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-4f278a78-f808-4cf1-a2e6-c27cc8a2f048 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0996205223344abfa89c41bd1a033342', 'request_id': 'req-4f278a78-f808-4cf1-a2e6-c27cc8a2f048'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:51.845106 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-4f278a78-f808-4cf1-a2e6-c27cc8a2f048 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0996205223344abfa89c41bd1a033342', 'request_id': 'req-4f278a78-f808-4cf1-a2e6-c27cc8a2f048'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:51.852487 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-4f278a78-f808-4cf1-a2e6-c27cc8a2f048 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0996205223344abfa89c41bd1a033342', 'request_id': 'req-4f278a78-f808-4cf1-a2e6-c27cc8a2f048'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:51.868353 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-4f278a78-f808-4cf1-a2e6-c27cc8a2f048 admin admin] The token cache is being invalidate because project 0996205223344abfa89c41bd1a033342 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:36:51.868353 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-4f278a78-f808-4cf1-a2e6-c27cc8a2f048 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:51.869132 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-4f278a78-f808-4cf1-a2e6-c27cc8a2f048 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:51.874510 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 939/1901] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:36:51 2022] DELETE /identity/v3/projects/0996205223344abfa89c41bd1a033342 => generated 0 bytes in 216 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:36:51.876202 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e85fee4-0d40-4d5c-8464-8190068c3d56 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:51.876577 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e85fee4-0d40-4d5c-8464-8190068c3d56 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:51.877090 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e85fee4-0d40-4d5c-8464-8190068c3d56 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:51.902240 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-4e85fee4-0d40-4d5c-8464-8190068c3d56 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:36:51.904299 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-4e85fee4-0d40-4d5c-8464-8190068c3d56 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:51.906634 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f667d34-1cb2-4fd4-901e-fd3096c0a462 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:51.907480 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f667d34-1cb2-4fd4-901e-fd3096c0a462 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:51.907901 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f667d34-1cb2-4fd4-901e-fd3096c0a462 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:51.908181 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f667d34-1cb2-4fd4-901e-fd3096c0a462 admin admin] PATH_INFO: `/v3/projects/bdfa54b3dbf4404f8c32c953d2b49e72` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:51.909326 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f667d34-1cb2-4fd4-901e-fd3096c0a462 admin admin] RBAC: Authorizing `identity:delete_project(project_id=bdfa54b3dbf4404f8c32c953d2b49e72)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:51.911461 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f667d34-1cb2-4fd4-901e-fd3096c0a462 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:51.916845 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 940/1902] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:36:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:36:51.925075 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f46863c-75d2-4f99-b56a-486436b15a59 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:51.925435 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f46863c-75d2-4f99-b56a-486436b15a59 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:51.925784 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f46863c-75d2-4f99-b56a-486436b15a59 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:51.948081 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-9f46863c-75d2-4f99-b56a-486436b15a59 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:36:51.949476 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-9f46863c-75d2-4f99-b56a-486436b15a59 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:51.958894 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:36:51.966072 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-9f667d34-1cb2-4fd4-901e-fd3096c0a462 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bdfa54b3dbf4404f8c32c953d2b49e72', 'request_id': 'req-9f667d34-1cb2-4fd4-901e-fd3096c0a462'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:51.975360 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-9f667d34-1cb2-4fd4-901e-fd3096c0a462 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bdfa54b3dbf4404f8c32c953d2b49e72', 'request_id': 'req-9f667d34-1cb2-4fd4-901e-fd3096c0a462'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:51.982849 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-9f667d34-1cb2-4fd4-901e-fd3096c0a462 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bdfa54b3dbf4404f8c32c953d2b49e72', 'request_id': 'req-9f667d34-1cb2-4fd4-901e-fd3096c0a462'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:51.994401 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-9f667d34-1cb2-4fd4-901e-fd3096c0a462 admin admin] The token cache is being invalidate because project bdfa54b3dbf4404f8c32c953d2b49e72 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81434) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:36:51.994702 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-9f667d34-1cb2-4fd4-901e-fd3096c0a462 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:51.995353 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-9f667d34-1cb2-4fd4-901e-fd3096c0a462 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:51.998399 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 963/1903] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:36:51 2022] DELETE /identity/v3/projects/bdfa54b3dbf4404f8c32c953d2b49e72 => generated 0 bytes in 205 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:36:52.001968 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f667d34-1cb2-4fd4-901e-fd3096c0a462 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:52.003427 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-9f667d34-1cb2-4fd4-901e-fd3096c0a462 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:52.025642 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 941/1904] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:36:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:36:52.026711 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f46863c-75d2-4f99-b56a-486436b15a59 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:52.036481 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f46863c-75d2-4f99-b56a-486436b15a59 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:52.041306 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6d144e2-7974-4052-89e9-f2e8b67128a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:52.041919 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6d144e2-7974-4052-89e9-f2e8b67128a0 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:52.042201 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6d144e2-7974-4052-89e9-f2e8b67128a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:52.042471 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6d144e2-7974-4052-89e9-f2e8b67128a0 admin admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:52.044320 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6d144e2-7974-4052-89e9-f2e8b67128a0 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:36:52.044320 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6d144e2-7974-4052-89e9-f2e8b67128a0 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:52.046687 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6d144e2-7974-4052-89e9-f2e8b67128a0 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:52.048553 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:36:52.060570 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 942/1905] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Sep 25 04:36:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:36:52.064728 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6d144e2-7974-4052-89e9-f2e8b67128a0 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:52.073789 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f667d34-1cb2-4fd4-901e-fd3096c0a462 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:52.076606 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6d144e2-7974-4052-89e9-f2e8b67128a0 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:52.079330 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de2f1fd7-6c39-4dea-92f7-5712c571ba7e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:52.079981 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de2f1fd7-6c39-4dea-92f7-5712c571ba7e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:52.080294 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de2f1fd7-6c39-4dea-92f7-5712c571ba7e admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:52.080568 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de2f1fd7-6c39-4dea-92f7-5712c571ba7e admin admin] PATH_INFO: `/v3/users/b3ca94b4e4ec43a6892b52af62b0e477` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:52.081738 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de2f1fd7-6c39-4dea-92f7-5712c571ba7e admin admin] RBAC: Authorizing `identity:delete_user(user_id=b3ca94b4e4ec43a6892b52af62b0e477)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:52.083264 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db306bda-a84b-4880-b98a-02f053d69b4c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:52.083647 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db306bda-a84b-4880-b98a-02f053d69b4c admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:52.083924 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de2f1fd7-6c39-4dea-92f7-5712c571ba7e admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:52.084023 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db306bda-a84b-4880-b98a-02f053d69b4c admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:52.084378 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db306bda-a84b-4880-b98a-02f053d69b4c admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:52.085301 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db306bda-a84b-4880-b98a-02f053d69b4c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:52.087880 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db306bda-a84b-4880-b98a-02f053d69b4c admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:52.136705 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 943/1906] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:36:52 2022] POST /identity/v3/projects => generated 385 bytes in 73 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 04:36:52.140986 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-de2f1fd7-6c39-4dea-92f7-5712c571ba7e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b3ca94b4e4ec43a6892b52af62b0e477'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:52.142630 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db306bda-a84b-4880-b98a-02f053d69b4c admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:52.149147 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-de2f1fd7-6c39-4dea-92f7-5712c571ba7e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b3ca94b4e4ec43a6892b52af62b0e477'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:52.167705 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-de2f1fd7-6c39-4dea-92f7-5712c571ba7e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b3ca94b4e4ec43a6892b52af62b0e477'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:52.169249 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-de2f1fd7-6c39-4dea-92f7-5712c571ba7e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b3ca94b4e4ec43a6892b52af62b0e477'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:52.177730 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-de2f1fd7-6c39-4dea-92f7-5712c571ba7e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b3ca94b4e4ec43a6892b52af62b0e477'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:52.188740 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 964/1907] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:36:52 2022] DELETE /identity/v3/users/b3ca94b4e4ec43a6892b52af62b0e477 => generated 0 bytes in 184 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:36:52.233931 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db306bda-a84b-4880-b98a-02f053d69b4c admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:52.308051 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75b021d1-1d27-4aae-97a7-765483ab6cb6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:52.308798 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75b021d1-1d27-4aae-97a7-765483ab6cb6 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:52.309614 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75b021d1-1d27-4aae-97a7-765483ab6cb6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:52.309614 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75b021d1-1d27-4aae-97a7-765483ab6cb6 admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:52.310312 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75b021d1-1d27-4aae-97a7-765483ab6cb6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:52.312306 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75b021d1-1d27-4aae-97a7-765483ab6cb6 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:52.336288 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 944/1908] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:36:52 2022] POST /identity/v3/users => generated 293 bytes in 195 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 04:36:52.340912 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de2f1fd7-6c39-4dea-92f7-5712c571ba7e admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:52.342366 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-de2f1fd7-6c39-4dea-92f7-5712c571ba7e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:52.357131 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de2f1fd7-6c39-4dea-92f7-5712c571ba7e admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:52.362989 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3457ec9-1d76-4f23-9308-1dcb9e81a919 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:52.363664 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3457ec9-1d76-4f23-9308-1dcb9e81a919 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:52.363959 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3457ec9-1d76-4f23-9308-1dcb9e81a919 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:52.364235 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3457ec9-1d76-4f23-9308-1dcb9e81a919 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:52.364944 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3457ec9-1d76-4f23-9308-1dcb9e81a919 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:52.366860 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3457ec9-1d76-4f23-9308-1dcb9e81a919 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:52.382265 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75b021d1-1d27-4aae-97a7-765483ab6cb6 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:52.383006 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 965/1909] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:36:52 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:36:52.394052 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75b021d1-1d27-4aae-97a7-765483ab6cb6 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:52.399702 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e97ef3c9-303e-44c2-868b-a090f32975c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:52.400404 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e97ef3c9-303e-44c2-868b-a090f32975c4 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:52.400815 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e97ef3c9-303e-44c2-868b-a090f32975c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:52.401166 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e97ef3c9-303e-44c2-868b-a090f32975c4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:52.401878 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e97ef3c9-303e-44c2-868b-a090f32975c4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:52.404245 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e97ef3c9-303e-44c2-868b-a090f32975c4 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:52.414531 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 945/1910] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:36:52 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:36:52.418993 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3457ec9-1d76-4f23-9308-1dcb9e81a919 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:52.429489 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3457ec9-1d76-4f23-9308-1dcb9e81a919 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:52.435015 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46337392-0730-441c-9e77-7ee3fd5f979b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:52.435785 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46337392-0730-441c-9e77-7ee3fd5f979b admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:52.436066 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46337392-0730-441c-9e77-7ee3fd5f979b admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:52.436367 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46337392-0730-441c-9e77-7ee3fd5f979b admin admin] PATH_INFO: `/v3/projects/d76a809856fb44f68a3a7358ba8ad381/users/1b22f735f34645f5b2ed7aa7b0489d48/roles/8f6cc574056c4278bc9778f14dc4b411` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:52.452600 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46337392-0730-441c-9e77-7ee3fd5f979b admin admin] RBAC: Authorizing `identity:create_grant(project_id=d76a809856fb44f68a3a7358ba8ad381, user_id=1b22f735f34645f5b2ed7aa7b0489d48, role_id=8f6cc574056c4278bc9778f14dc4b411)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:52.455267 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46337392-0730-441c-9e77-7ee3fd5f979b admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:52.470999 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 966/1911] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:36:52 2022] PUT /identity/v3/projects/d76a809856fb44f68a3a7358ba8ad381/users/1b22f735f34645f5b2ed7aa7b0489d48/roles/8f6cc574056c4278bc9778f14dc4b411 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:36:52.481529 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90f609ce-a460-47e8-9769-6e857530ad96 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:52.482143 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90f609ce-a460-47e8-9769-6e857530ad96 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:52.482527 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90f609ce-a460-47e8-9769-6e857530ad96 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:52.513732 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-90f609ce-a460-47e8-9769-6e857530ad96 None None] MFA Rules not processed for user `1b22f735f34645f5b2ed7aa7b0489d48`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:36:52.515010 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-90f609ce-a460-47e8-9769-6e857530ad96 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:52.563881 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46337392-0730-441c-9e77-7ee3fd5f979b admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:52.565777 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-46337392-0730-441c-9e77-7ee3fd5f979b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:52.569583 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:36:52.588459 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=83,keystone:DELETE=14,keystone:INSERT=3 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:36:52.591302 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 946/1912] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:36:52 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:36:52.595753 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90f609ce-a460-47e8-9769-6e857530ad96 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:52.647651 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46337392-0730-441c-9e77-7ee3fd5f979b admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:52.652529 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fab82a8b-dc7e-4592-834b-8c0f2ab42baa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:52.653415 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fab82a8b-dc7e-4592-834b-8c0f2ab42baa admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:52.653881 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fab82a8b-dc7e-4592-834b-8c0f2ab42baa admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:52.654159 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fab82a8b-dc7e-4592-834b-8c0f2ab42baa admin admin] PATH_INFO: `/v3/projects/740000af86e94b97b973dd68f26c4d2d` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:52.655449 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fab82a8b-dc7e-4592-834b-8c0f2ab42baa admin admin] RBAC: Authorizing `identity:delete_project(project_id=740000af86e94b97b973dd68f26c4d2d)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:52.657451 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fab82a8b-dc7e-4592-834b-8c0f2ab42baa admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:52.661634 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90f609ce-a460-47e8-9769-6e857530ad96 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:52.666676 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c77ac246-b5c2-42da-a05b-a0f043be2c2a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:52.667285 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c77ac246-b5c2-42da-a05b-a0f043be2c2a admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:52.667561 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c77ac246-b5c2-42da-a05b-a0f043be2c2a admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:52.667839 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c77ac246-b5c2-42da-a05b-a0f043be2c2a admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:52.669055 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c77ac246-b5c2-42da-a05b-a0f043be2c2a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:52.671204 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c77ac246-b5c2-42da-a05b-a0f043be2c2a admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:52.698724 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 947/1913] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:36:52 2022] POST /identity/v3/projects => generated 383 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 04:36:52.703281 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c77ac246-b5c2-42da-a05b-a0f043be2c2a admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:52.706471 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-fab82a8b-dc7e-4592-834b-8c0f2ab42baa admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '740000af86e94b97b973dd68f26c4d2d', 'request_id': 'req-fab82a8b-dc7e-4592-834b-8c0f2ab42baa'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:52.716245 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-fab82a8b-dc7e-4592-834b-8c0f2ab42baa admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '740000af86e94b97b973dd68f26c4d2d', 'request_id': 'req-fab82a8b-dc7e-4592-834b-8c0f2ab42baa'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:52.722704 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-fab82a8b-dc7e-4592-834b-8c0f2ab42baa admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '740000af86e94b97b973dd68f26c4d2d', 'request_id': 'req-fab82a8b-dc7e-4592-834b-8c0f2ab42baa'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:52.734702 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-fab82a8b-dc7e-4592-834b-8c0f2ab42baa admin admin] The token cache is being invalidate because project 740000af86e94b97b973dd68f26c4d2d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81434) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:36:52.734995 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-fab82a8b-dc7e-4592-834b-8c0f2ab42baa admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:52.737074 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-fab82a8b-dc7e-4592-834b-8c0f2ab42baa admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:52.740626 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 967/1914] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:36:52 2022] DELETE /identity/v3/projects/740000af86e94b97b973dd68f26c4d2d => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 04:36:52.741868 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9ea3f9d-6f1b-4a84-a7fb-5f4af90c6b68 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:52.742149 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9ea3f9d-6f1b-4a84-a7fb-5f4af90c6b68 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:52.742420 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9ea3f9d-6f1b-4a84-a7fb-5f4af90c6b68 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:52.764771 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-d9ea3f9d-6f1b-4a84-a7fb-5f4af90c6b68 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:36:52.765979 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-d9ea3f9d-6f1b-4a84-a7fb-5f4af90c6b68 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:52.781747 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c77ac246-b5c2-42da-a05b-a0f043be2c2a admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:52.788356 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-095eaf63-e7d4-42a8-973d-738852ca201e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:52.789209 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-095eaf63-e7d4-42a8-973d-738852ca201e admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:52.789560 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-095eaf63-e7d4-42a8-973d-738852ca201e admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:52.789901 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-095eaf63-e7d4-42a8-973d-738852ca201e admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:52.790694 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-095eaf63-e7d4-42a8-973d-738852ca201e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:52.792923 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-095eaf63-e7d4-42a8-973d-738852ca201e admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:52.818573 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 948/1915] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:36:52 2022] POST /identity/v3/users => generated 292 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:36:52.822325 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-095eaf63-e7d4-42a8-973d-738852ca201e admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:52.824272 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-095eaf63-e7d4-42a8-973d-738852ca201e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:52.834100 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 968/1916] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:36:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 04:36:52.841343 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-095eaf63-e7d4-42a8-973d-738852ca201e admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:52.841840 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-910dc459-1a27-4832-84fe-4949192b8bc7 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:52.842220 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-910dc459-1a27-4832-84fe-4949192b8bc7 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:52.842412 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-910dc459-1a27-4832-84fe-4949192b8bc7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:52.846685 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2c9232c-158d-4426-a7b9-76d1ad8ff956 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:52.847440 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c9232c-158d-4426-a7b9-76d1ad8ff956 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:52.847795 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c9232c-158d-4426-a7b9-76d1ad8ff956 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:52.848137 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c9232c-158d-4426-a7b9-76d1ad8ff956 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:52.848896 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2c9232c-158d-4426-a7b9-76d1ad8ff956 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:52.851630 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2c9232c-158d-4426-a7b9-76d1ad8ff956 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:52.863497 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-910dc459-1a27-4832-84fe-4949192b8bc7 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:36:52.865112 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 949/1917] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:36:52 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 25 04:36:52.865253 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-910dc459-1a27-4832-84fe-4949192b8bc7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:52.868514 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2c9232c-158d-4426-a7b9-76d1ad8ff956 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:52.873629 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 969/1918] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:36:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 04:36:52.877586 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-910dc459-1a27-4832-84fe-4949192b8bc7 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:52.879404 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2c9232c-158d-4426-a7b9-76d1ad8ff956 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:52.884298 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04766e6b-af90-4401-8a50-d07712dae2e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:52.884991 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04766e6b-af90-4401-8a50-d07712dae2e1 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:52.885381 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04766e6b-af90-4401-8a50-d07712dae2e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:52.885766 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04766e6b-af90-4401-8a50-d07712dae2e1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:52.886572 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04766e6b-af90-4401-8a50-d07712dae2e1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:52.888852 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04766e6b-af90-4401-8a50-d07712dae2e1 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:52.888960 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-910dc459-1a27-4832-84fe-4949192b8bc7 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:52.904661 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-776ec3b0-1b9f-41ee-8039-804e03c1d3e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:52.904661 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-776ec3b0-1b9f-41ee-8039-804e03c1d3e1 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:52.904661 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-776ec3b0-1b9f-41ee-8039-804e03c1d3e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:52.904661 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-776ec3b0-1b9f-41ee-8039-804e03c1d3e1 admin admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:52.905232 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-776ec3b0-1b9f-41ee-8039-804e03c1d3e1 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:36:52.905232 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-776ec3b0-1b9f-41ee-8039-804e03c1d3e1 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:52.905232 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-776ec3b0-1b9f-41ee-8039-804e03c1d3e1 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:52.908997 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 950/1919] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:36:52 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:36:52.910388 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 970/1920] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Sep 25 04:36:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:36:52.913741 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04766e6b-af90-4401-8a50-d07712dae2e1 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:52.914418 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-776ec3b0-1b9f-41ee-8039-804e03c1d3e1 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:52.924222 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04766e6b-af90-4401-8a50-d07712dae2e1 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:52.924964 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-776ec3b0-1b9f-41ee-8039-804e03c1d3e1 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:52.929008 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e55535d-c81c-4778-848d-82e7b07f810b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:52.929665 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e55535d-c81c-4778-848d-82e7b07f810b admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:52.929780 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0584511a-8941-433e-b4fc-7266cadbb2da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:52.929918 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e55535d-c81c-4778-848d-82e7b07f810b admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:52.930189 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e55535d-c81c-4778-848d-82e7b07f810b admin admin] PATH_INFO: `/v3/projects/304070002d394b50b7aa0a45d9512572/users/98be5bb31c954612a8eb3fe2ab8041b5/roles/8f6cc574056c4278bc9778f14dc4b411` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:52.930399 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0584511a-8941-433e-b4fc-7266cadbb2da admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:52.930682 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0584511a-8941-433e-b4fc-7266cadbb2da admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:52.930956 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0584511a-8941-433e-b4fc-7266cadbb2da admin admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:52.931587 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0584511a-8941-433e-b4fc-7266cadbb2da admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:52.933553 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0584511a-8941-433e-b4fc-7266cadbb2da admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:52.945620 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e55535d-c81c-4778-848d-82e7b07f810b admin admin] RBAC: Authorizing `identity:create_grant(project_id=304070002d394b50b7aa0a45d9512572, user_id=98be5bb31c954612a8eb3fe2ab8041b5, role_id=8f6cc574056c4278bc9778f14dc4b411)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:52.948250 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e55535d-c81c-4778-848d-82e7b07f810b admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:52.964799 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 971/1921] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:36:52 2022] POST /identity/v3/projects => generated 381 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 04:36:52.965019 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 951/1922] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:36:52 2022] PUT /identity/v3/projects/304070002d394b50b7aa0a45d9512572/users/98be5bb31c954612a8eb3fe2ab8041b5/roles/8f6cc574056c4278bc9778f14dc4b411 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:36:52.970257 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0584511a-8941-433e-b4fc-7266cadbb2da admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:52.973161 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c506b4-deef-4bcb-8869-901be81e5beb None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:52.973415 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c506b4-deef-4bcb-8869-901be81e5beb None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:52.973704 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c506b4-deef-4bcb-8869-901be81e5beb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:52.974728 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:36:53.002732 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-15c506b4-deef-4bcb-8869-901be81e5beb None None] MFA Rules not processed for user `98be5bb31c954612a8eb3fe2ab8041b5`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:36:53.003985 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-15c506b4-deef-4bcb-8869-901be81e5beb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:53.009147 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:36:53.037456 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0584511a-8941-433e-b4fc-7266cadbb2da admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:53.042181 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f6acd4b-d0bd-4913-ad2f-587cbf4b584f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:53.042795 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f6acd4b-d0bd-4913-ad2f-587cbf4b584f admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:53.043078 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f6acd4b-d0bd-4913-ad2f-587cbf4b584f admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:53.043350 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f6acd4b-d0bd-4913-ad2f-587cbf4b584f admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:53.044067 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f6acd4b-d0bd-4913-ad2f-587cbf4b584f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:53.046021 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f6acd4b-d0bd-4913-ad2f-587cbf4b584f admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:53.071877 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 972/1923] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:36:52 2022] POST /identity/v3/users => generated 291 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:36:53.073020 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 952/1924] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:36:52 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 04:36:53.073352 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f6acd4b-d0bd-4913-ad2f-587cbf4b584f admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:53.077868 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15c506b4-deef-4bcb-8869-901be81e5beb None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:53.084123 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f6acd4b-d0bd-4913-ad2f-587cbf4b584f admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:53.088227 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15c506b4-deef-4bcb-8869-901be81e5beb None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:53.088805 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b72afd5-c0ed-443b-8f91-fce424b20620 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:53.089402 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b72afd5-c0ed-443b-8f91-fce424b20620 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:53.089683 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b72afd5-c0ed-443b-8f91-fce424b20620 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:53.089961 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b72afd5-c0ed-443b-8f91-fce424b20620 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:53.090631 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b72afd5-c0ed-443b-8f91-fce424b20620 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:53.092223 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b72afd5-c0ed-443b-8f91-fce424b20620 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:53.093093 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-101e13f6-0d84-4090-9786-4b8dd540b33f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:53.093679 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-101e13f6-0d84-4090-9786-4b8dd540b33f admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:53.093984 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-101e13f6-0d84-4090-9786-4b8dd540b33f admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:53.094254 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-101e13f6-0d84-4090-9786-4b8dd540b33f admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:53.094879 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-101e13f6-0d84-4090-9786-4b8dd540b33f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:53.096777 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-101e13f6-0d84-4090-9786-4b8dd540b33f admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:53.102232 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 973/1925] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:36:53 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 25 04:36:53.106334 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b72afd5-c0ed-443b-8f91-fce424b20620 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:53.116207 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b72afd5-c0ed-443b-8f91-fce424b20620 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:53.122405 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16e361b7-1179-4554-833c-4732db51fb0c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:53.123020 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16e361b7-1179-4554-833c-4732db51fb0c admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:53.123323 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16e361b7-1179-4554-833c-4732db51fb0c admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:53.123542 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16e361b7-1179-4554-833c-4732db51fb0c admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:53.124176 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16e361b7-1179-4554-833c-4732db51fb0c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:53.125914 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16e361b7-1179-4554-833c-4732db51fb0c admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:53.131979 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 953/1926] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:36:53 2022] POST /identity/v3/projects => generated 385 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 04:36:53.135781 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-101e13f6-0d84-4090-9786-4b8dd540b33f admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:53.142053 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 974/1927] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:36:53 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 25 04:36:53.146625 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16e361b7-1179-4554-833c-4732db51fb0c admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:53.161338 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16e361b7-1179-4554-833c-4732db51fb0c admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:53.213219 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-101e13f6-0d84-4090-9786-4b8dd540b33f admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:53.223319 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89d52d5b-c435-4136-a238-a4ced75e3479 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:53.223870 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d52d5b-c435-4136-a238-a4ced75e3479 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:53.224149 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d52d5b-c435-4136-a238-a4ced75e3479 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:53.224420 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d52d5b-c435-4136-a238-a4ced75e3479 admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:53.225119 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89d52d5b-c435-4136-a238-a4ced75e3479 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:53.225556 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89f82f8f-ae58-46c3-897d-cec9071bdadc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:53.226242 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89f82f8f-ae58-46c3-897d-cec9071bdadc admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:53.226530 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89f82f8f-ae58-46c3-897d-cec9071bdadc admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:53.226808 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89f82f8f-ae58-46c3-897d-cec9071bdadc admin admin] PATH_INFO: `/v3/projects/07a6ee99f14c4e5aaef0619b0ad0858e/users/1b9ce416d14e4d2c80ef7387a65ab17e/roles/8f6cc574056c4278bc9778f14dc4b411` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:53.227139 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89d52d5b-c435-4136-a238-a4ced75e3479 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:53.243263 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89f82f8f-ae58-46c3-897d-cec9071bdadc admin admin] RBAC: Authorizing `identity:create_grant(project_id=07a6ee99f14c4e5aaef0619b0ad0858e, user_id=1b9ce416d14e4d2c80ef7387a65ab17e, role_id=8f6cc574056c4278bc9778f14dc4b411)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:53.246344 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89f82f8f-ae58-46c3-897d-cec9071bdadc admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:53.255560 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 954/1928] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:36:53 2022] POST /identity/v3/users => generated 299 bytes in 120 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 04:36:53.259692 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89d52d5b-c435-4136-a238-a4ced75e3479 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:53.264769 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 975/1929] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:36:53 2022] PUT /identity/v3/projects/07a6ee99f14c4e5aaef0619b0ad0858e/users/1b9ce416d14e4d2c80ef7387a65ab17e/roles/8f6cc574056c4278bc9778f14dc4b411 => generated 0 bytes in 119 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:36:53.269722 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89d52d5b-c435-4136-a238-a4ced75e3479 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:53.275798 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-722839d9-0d35-4bd4-896b-724767c4abb2 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:53.276671 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-722839d9-0d35-4bd4-896b-724767c4abb2 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:53.276671 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-722839d9-0d35-4bd4-896b-724767c4abb2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:53.306569 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-722839d9-0d35-4bd4-896b-724767c4abb2 None None] MFA Rules not processed for user `1b9ce416d14e4d2c80ef7387a65ab17e`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:36:53.307788 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-722839d9-0d35-4bd4-896b-724767c4abb2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:53.340440 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-857e09f3-66fb-45be-9956-1a37be83c0cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:53.341422 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-857e09f3-66fb-45be-9956-1a37be83c0cb admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:53.341705 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-857e09f3-66fb-45be-9956-1a37be83c0cb admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:53.341979 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-857e09f3-66fb-45be-9956-1a37be83c0cb admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:53.342668 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-857e09f3-66fb-45be-9956-1a37be83c0cb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:53.344217 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-857e09f3-66fb-45be-9956-1a37be83c0cb admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:53.354975 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 955/1930] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:36:53 2022] GET /identity/v3/roles => generated 1420 bytes in 96 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 25 04:36:53.359408 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-857e09f3-66fb-45be-9956-1a37be83c0cb admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:53.367834 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:36:53.372346 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-857e09f3-66fb-45be-9956-1a37be83c0cb admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:53.378083 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2144b297-7c15-418e-b644-baed4937d80d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:53.378756 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2144b297-7c15-418e-b644-baed4937d80d admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:53.379041 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2144b297-7c15-418e-b644-baed4937d80d admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:53.379318 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2144b297-7c15-418e-b644-baed4937d80d admin admin] PATH_INFO: `/v3/projects/85141bb6e8944b2a8deda483fb0d4352/users/b9464098dc944ab7bb55ec08742e128e/roles/c51fe7c5ad804040879acfb54269455f` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:53.389152 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:36:53.397645 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 976/1931] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:36:53 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:36:53.398129 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2144b297-7c15-418e-b644-baed4937d80d admin admin] RBAC: Authorizing `identity:create_grant(project_id=85141bb6e8944b2a8deda483fb0d4352, user_id=b9464098dc944ab7bb55ec08742e128e, role_id=c51fe7c5ad804040879acfb54269455f)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:53.400905 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2144b297-7c15-418e-b644-baed4937d80d admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:53.403323 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-722839d9-0d35-4bd4-896b-724767c4abb2 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:53.415349 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-722839d9-0d35-4bd4-896b-724767c4abb2 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:53.417751 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 956/1932] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:36:53 2022] PUT /identity/v3/projects/85141bb6e8944b2a8deda483fb0d4352/users/b9464098dc944ab7bb55ec08742e128e/roles/c51fe7c5ad804040879acfb54269455f => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 04:36:53.422581 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2144b297-7c15-418e-b644-baed4937d80d admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:53.489412 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d32e269-4d4f-4d6d-80d5-4062806c18d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:53.490707 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d32e269-4d4f-4d6d-80d5-4062806c18d9 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:53.490995 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d32e269-4d4f-4d6d-80d5-4062806c18d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:53.491264 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d32e269-4d4f-4d6d-80d5-4062806c18d9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:53.491902 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d32e269-4d4f-4d6d-80d5-4062806c18d9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:53.493913 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2144b297-7c15-418e-b644-baed4937d80d admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:53.498069 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d32e269-4d4f-4d6d-80d5-4062806c18d9 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:53.499873 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b00d94af-b4dc-4eb3-9334-da5132fc964d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:53.500487 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b00d94af-b4dc-4eb3-9334-da5132fc964d admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:53.500809 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b00d94af-b4dc-4eb3-9334-da5132fc964d admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:53.501136 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b00d94af-b4dc-4eb3-9334-da5132fc964d admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:53.501721 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b00d94af-b4dc-4eb3-9334-da5132fc964d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:53.503302 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b00d94af-b4dc-4eb3-9334-da5132fc964d admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:53.514591 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 957/1933] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:36:53 2022] GET /identity/v3/roles => generated 1420 bytes in 92 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 25 04:36:53.518589 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b00d94af-b4dc-4eb3-9334-da5132fc964d admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:53.530326 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b00d94af-b4dc-4eb3-9334-da5132fc964d admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:53.533077 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 977/1934] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:36:53 2022] POST /identity/v3/projects => generated 379 bytes in 130 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 04:36:53.536981 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d32e269-4d4f-4d6d-80d5-4062806c18d9 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:53.538365 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e849a25-eb0d-4860-b1a7-82a1e075146a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:53.539039 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e849a25-eb0d-4860-b1a7-82a1e075146a admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:53.539322 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e849a25-eb0d-4860-b1a7-82a1e075146a admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:53.539599 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e849a25-eb0d-4860-b1a7-82a1e075146a admin admin] PATH_INFO: `/v3/projects/85141bb6e8944b2a8deda483fb0d4352/users/b9464098dc944ab7bb55ec08742e128e/roles/c51fe7c5ad804040879acfb54269455f` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:53.541619 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e849a25-eb0d-4860-b1a7-82a1e075146a admin admin] RBAC: Authorizing `identity:create_grant(project_id=85141bb6e8944b2a8deda483fb0d4352, user_id=b9464098dc944ab7bb55ec08742e128e, role_id=c51fe7c5ad804040879acfb54269455f)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:53.545304 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e849a25-eb0d-4860-b1a7-82a1e075146a admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:53.556918 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 958/1935] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:36:53 2022] PUT /identity/v3/projects/85141bb6e8944b2a8deda483fb0d4352/users/b9464098dc944ab7bb55ec08742e128e/roles/c51fe7c5ad804040879acfb54269455f => generated 0 bytes in 39 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:36:53.567406 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fddbafbe-0b54-4525-945f-86224b5b9517 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:53.568210 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fddbafbe-0b54-4525-945f-86224b5b9517 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:53.568210 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fddbafbe-0b54-4525-945f-86224b5b9517 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:53.598921 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-fddbafbe-0b54-4525-945f-86224b5b9517 None None] MFA Rules not processed for user `b9464098dc944ab7bb55ec08742e128e`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:36:53.600206 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-fddbafbe-0b54-4525-945f-86224b5b9517 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:53.617032 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d32e269-4d4f-4d6d-80d5-4062806c18d9 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:53.679595 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 959/1936] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:36:53 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:36:53.686741 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aee303a-8389-484a-b4b1-aea90bbb51f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:53.687481 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aee303a-8389-484a-b4b1-aea90bbb51f3 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:53.687863 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aee303a-8389-484a-b4b1-aea90bbb51f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:53.688142 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aee303a-8389-484a-b4b1-aea90bbb51f3 admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:53.688889 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aee303a-8389-484a-b4b1-aea90bbb51f3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:53.690890 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aee303a-8389-484a-b4b1-aea90bbb51f3 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:53.698646 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fddbafbe-0b54-4525-945f-86224b5b9517 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:53.700596 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-fddbafbe-0b54-4525-945f-86224b5b9517 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:53.716694 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 978/1937] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:36:53 2022] POST /identity/v3/users => generated 296 bytes in 177 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 04:36:53.718112 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aee303a-8389-484a-b4b1-aea90bbb51f3 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:53.724274 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:36:53.728269 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aee303a-8389-484a-b4b1-aea90bbb51f3 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:53.734682 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f93caa4-6330-4fed-bc3b-13a53861a98d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:53.735295 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f93caa4-6330-4fed-bc3b-13a53861a98d admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:53.735576 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f93caa4-6330-4fed-bc3b-13a53861a98d admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:53.735853 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f93caa4-6330-4fed-bc3b-13a53861a98d admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:53.736491 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f93caa4-6330-4fed-bc3b-13a53861a98d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:53.738295 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f93caa4-6330-4fed-bc3b-13a53861a98d admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:53.748343 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 979/1938] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:36:53 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 25 04:36:53.752362 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f93caa4-6330-4fed-bc3b-13a53861a98d admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:53.762251 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f93caa4-6330-4fed-bc3b-13a53861a98d admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:53.764373 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fddbafbe-0b54-4525-945f-86224b5b9517 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:53.767013 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f66f03c7-38c3-4979-9f55-c4841d14dc9c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:53.767661 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f66f03c7-38c3-4979-9f55-c4841d14dc9c admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:53.767945 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f66f03c7-38c3-4979-9f55-c4841d14dc9c admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:53.768065 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c32fe8c2-753f-45ac-8fe5-bb8c2d6a06df service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ffbc136c77b84e8c87128f7247f4ec03', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:53.768224 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f66f03c7-38c3-4979-9f55-c4841d14dc9c admin admin] PATH_INFO: `/v3/projects/da8d5bbb0c394e719169742c6a7e3818/users/202d4ffe4d5447e5b1342711b599d884/roles/c51fe7c5ad804040879acfb54269455f` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:53.768656 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c32fe8c2-753f-45ac-8fe5-bb8c2d6a06df service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:53.768961 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c32fe8c2-753f-45ac-8fe5-bb8c2d6a06df service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:53.769236 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c32fe8c2-753f-45ac-8fe5-bb8c2d6a06df service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:53.771582 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-c32fe8c2-753f-45ac-8fe5-bb8c2d6a06df service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:53.781334 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:36:53.784457 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f66f03c7-38c3-4979-9f55-c4841d14dc9c admin admin] RBAC: Authorizing `identity:create_grant(project_id=da8d5bbb0c394e719169742c6a7e3818, user_id=202d4ffe4d5447e5b1342711b599d884, role_id=c51fe7c5ad804040879acfb54269455f)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:53.787303 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f66f03c7-38c3-4979-9f55-c4841d14dc9c admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:53.800948 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 980/1939] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:36:53 2022] PUT /identity/v3/projects/da8d5bbb0c394e719169742c6a7e3818/users/202d4ffe4d5447e5b1342711b599d884/roles/c51fe7c5ad804040879acfb54269455f => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 04:36:53.804846 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f66f03c7-38c3-4979-9f55-c4841d14dc9c admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:53.834295 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c32fe8c2-753f-45ac-8fe5-bb8c2d6a06df service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:53.835736 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c32fe8c2-753f-45ac-8fe5-bb8c2d6a06df service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:53.845281 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 960/1940] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:36:53 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 147 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 25 04:36:53.866447 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f66f03c7-38c3-4979-9f55-c4841d14dc9c admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:53.872615 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10410115-a6d7-45c2-9145-9ca0ed0b58cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:53.873257 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10410115-a6d7-45c2-9145-9ca0ed0b58cd admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:53.873545 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10410115-a6d7-45c2-9145-9ca0ed0b58cd admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:53.873823 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10410115-a6d7-45c2-9145-9ca0ed0b58cd admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:53.874468 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10410115-a6d7-45c2-9145-9ca0ed0b58cd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:53.876067 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10410115-a6d7-45c2-9145-9ca0ed0b58cd admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:53.886230 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 981/1941] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:36:53 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 25 04:36:53.890556 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c32fe8c2-753f-45ac-8fe5-bb8c2d6a06df service cinder] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:53.901559 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c32fe8c2-753f-45ac-8fe5-bb8c2d6a06df service cinder] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:53.907424 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfa3015b-72d5-4257-b04b-6f27e81f0805 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:53.908165 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfa3015b-72d5-4257-b04b-6f27e81f0805 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:53.908542 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfa3015b-72d5-4257-b04b-6f27e81f0805 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:53.908857 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfa3015b-72d5-4257-b04b-6f27e81f0805 admin admin] PATH_INFO: `/v3/projects/da8d5bbb0c394e719169742c6a7e3818/users/202d4ffe4d5447e5b1342711b599d884/roles/c51fe7c5ad804040879acfb54269455f` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:53.911029 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfa3015b-72d5-4257-b04b-6f27e81f0805 admin admin] RBAC: Authorizing `identity:create_grant(project_id=da8d5bbb0c394e719169742c6a7e3818, user_id=202d4ffe4d5447e5b1342711b599d884, role_id=c51fe7c5ad804040879acfb54269455f)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:53.913785 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfa3015b-72d5-4257-b04b-6f27e81f0805 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:53.930480 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 961/1942] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:36:53 2022] PUT /identity/v3/projects/da8d5bbb0c394e719169742c6a7e3818/users/202d4ffe4d5447e5b1342711b599d884/roles/c51fe7c5ad804040879acfb54269455f => generated 0 bytes in 40 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:36:53.943157 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a535c0ae-c9ab-45b5-8351-c60c710e8004 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:53.944612 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a535c0ae-c9ab-45b5-8351-c60c710e8004 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:53.944612 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a535c0ae-c9ab-45b5-8351-c60c710e8004 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:53.979538 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-a535c0ae-c9ab-45b5-8351-c60c710e8004 None None] MFA Rules not processed for user `202d4ffe4d5447e5b1342711b599d884`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:36:53.980830 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-a535c0ae-c9ab-45b5-8351-c60c710e8004 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:54.091397 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 982/1943] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:36:53 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 04:36:54.104974 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfa3015b-72d5-4257-b04b-6f27e81f0805 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:54.108438 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-cfa3015b-72d5-4257-b04b-6f27e81f0805 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:54.219586 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfa3015b-72d5-4257-b04b-6f27e81f0805 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:54.224093 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-425b1d8c-b80a-4468-bafe-c4b4e4b35abd service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ffbc136c77b84e8c87128f7247f4ec03', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:54.224744 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-425b1d8c-b80a-4468-bafe-c4b4e4b35abd service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:54.225742 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-425b1d8c-b80a-4468-bafe-c4b4e4b35abd service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:54.225742 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-425b1d8c-b80a-4468-bafe-c4b4e4b35abd service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:54.237243 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-425b1d8c-b80a-4468-bafe-c4b4e4b35abd service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:54.238999 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-425b1d8c-b80a-4468-bafe-c4b4e4b35abd service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:54.251128 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 962/1944] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:36:54 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 147 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:36:54.643691 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a535c0ae-c9ab-45b5-8351-c60c710e8004 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:54.653453 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a535c0ae-c9ab-45b5-8351-c60c710e8004 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:54.658515 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3e56b62-f76c-44cf-aad1-cf85d9913fb1 tempest-VolumeQuotaClassesTest-790700512 tempest-VolumeQuotaClassesTest-790700512-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '202d4ffe4d5447e5b1342711b599d884', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'da8d5bbb0c394e719169742c6a7e3818', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:54.659132 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3e56b62-f76c-44cf-aad1-cf85d9913fb1 tempest-VolumeQuotaClassesTest-790700512 tempest-VolumeQuotaClassesTest-790700512-project-admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:54.659411 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3e56b62-f76c-44cf-aad1-cf85d9913fb1 tempest-VolumeQuotaClassesTest-790700512 tempest-VolumeQuotaClassesTest-790700512-project-admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:54.659683 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3e56b62-f76c-44cf-aad1-cf85d9913fb1 tempest-VolumeQuotaClassesTest-790700512 tempest-VolumeQuotaClassesTest-790700512-project-admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:54.660325 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3e56b62-f76c-44cf-aad1-cf85d9913fb1 tempest-VolumeQuotaClassesTest-790700512 tempest-VolumeQuotaClassesTest-790700512-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:36:54.660627 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3e56b62-f76c-44cf-aad1-cf85d9913fb1 tempest-VolumeQuotaClassesTest-790700512 tempest-VolumeQuotaClassesTest-790700512-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:54.662256 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3e56b62-f76c-44cf-aad1-cf85d9913fb1 tempest-VolumeQuotaClassesTest-790700512 tempest-VolumeQuotaClassesTest-790700512-project-admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:54.679422 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 983/1945] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Sep 25 04:36:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:36:54.680499 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-425b1d8c-b80a-4468-bafe-c4b4e4b35abd service cinder] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:54.689902 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-425b1d8c-b80a-4468-bafe-c4b4e4b35abd service cinder] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:54.694523 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5187e0f1-fbff-434b-8a26-785d88998e51 tempest-VolumeQuotaClassesTest-790700512 tempest-VolumeQuotaClassesTest-790700512-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '202d4ffe4d5447e5b1342711b599d884', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'da8d5bbb0c394e719169742c6a7e3818', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:54.695157 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5187e0f1-fbff-434b-8a26-785d88998e51 tempest-VolumeQuotaClassesTest-790700512 tempest-VolumeQuotaClassesTest-790700512-project-admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:54.695442 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5187e0f1-fbff-434b-8a26-785d88998e51 tempest-VolumeQuotaClassesTest-790700512 tempest-VolumeQuotaClassesTest-790700512-project-admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:54.695718 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5187e0f1-fbff-434b-8a26-785d88998e51 tempest-VolumeQuotaClassesTest-790700512 tempest-VolumeQuotaClassesTest-790700512-project-admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:54.696362 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5187e0f1-fbff-434b-8a26-785d88998e51 tempest-VolumeQuotaClassesTest-790700512 tempest-VolumeQuotaClassesTest-790700512-project-admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:54.698308 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5187e0f1-fbff-434b-8a26-785d88998e51 tempest-VolumeQuotaClassesTest-790700512 tempest-VolumeQuotaClassesTest-790700512-project-admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:54.731657 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 963/1946] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:36:54 2022] POST /identity/v3/projects => generated 353 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:36:54.732868 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3e56b62-f76c-44cf-aad1-cf85d9913fb1 tempest-VolumeQuotaClassesTest-790700512 tempest-VolumeQuotaClassesTest-790700512-project-admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:54.742797 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3e56b62-f76c-44cf-aad1-cf85d9913fb1 tempest-VolumeQuotaClassesTest-790700512 tempest-VolumeQuotaClassesTest-790700512-project-admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:54.775441 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:36:54.801884 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-316dbf92-c7d1-481c-a18b-faf6d5105d07 tempest-VolumeQuotaClassesTest-790700512 tempest-VolumeQuotaClassesTest-790700512-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '202d4ffe4d5447e5b1342711b599d884', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'da8d5bbb0c394e719169742c6a7e3818', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:54.802615 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-316dbf92-c7d1-481c-a18b-faf6d5105d07 tempest-VolumeQuotaClassesTest-790700512 tempest-VolumeQuotaClassesTest-790700512-project-admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:54.802904 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-316dbf92-c7d1-481c-a18b-faf6d5105d07 tempest-VolumeQuotaClassesTest-790700512 tempest-VolumeQuotaClassesTest-790700512-project-admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:54.803183 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-316dbf92-c7d1-481c-a18b-faf6d5105d07 tempest-VolumeQuotaClassesTest-790700512 tempest-VolumeQuotaClassesTest-790700512-project-admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:54.803837 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-316dbf92-c7d1-481c-a18b-faf6d5105d07 tempest-VolumeQuotaClassesTest-790700512 tempest-VolumeQuotaClassesTest-790700512-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:36:54.804147 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-316dbf92-c7d1-481c-a18b-faf6d5105d07 tempest-VolumeQuotaClassesTest-790700512 tempest-VolumeQuotaClassesTest-790700512-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:54.805780 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-316dbf92-c7d1-481c-a18b-faf6d5105d07 tempest-VolumeQuotaClassesTest-790700512 tempest-VolumeQuotaClassesTest-790700512-project-admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:54.817931 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 984/1947] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Sep 25 04:36:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:36:54.861694 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5187e0f1-fbff-434b-8a26-785d88998e51 tempest-VolumeQuotaClassesTest-790700512 tempest-VolumeQuotaClassesTest-790700512-project-admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:54.872017 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5187e0f1-fbff-434b-8a26-785d88998e51 tempest-VolumeQuotaClassesTest-790700512 tempest-VolumeQuotaClassesTest-790700512-project-admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:54.876791 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-277ca1db-30b3-4f56-97a0-170444f5bdce tempest-VolumeQuotaClassesTest-790700512 tempest-VolumeQuotaClassesTest-790700512-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '202d4ffe4d5447e5b1342711b599d884', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'da8d5bbb0c394e719169742c6a7e3818', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:54.877443 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-277ca1db-30b3-4f56-97a0-170444f5bdce tempest-VolumeQuotaClassesTest-790700512 tempest-VolumeQuotaClassesTest-790700512-project-admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:54.877732 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-277ca1db-30b3-4f56-97a0-170444f5bdce tempest-VolumeQuotaClassesTest-790700512 tempest-VolumeQuotaClassesTest-790700512-project-admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:54.878009 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-277ca1db-30b3-4f56-97a0-170444f5bdce tempest-VolumeQuotaClassesTest-790700512 tempest-VolumeQuotaClassesTest-790700512-project-admin] PATH_INFO: `/v3/projects/a80ba44232be408b8df81737a22953d2` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:54.879131 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-277ca1db-30b3-4f56-97a0-170444f5bdce tempest-VolumeQuotaClassesTest-790700512 tempest-VolumeQuotaClassesTest-790700512-project-admin] RBAC: Authorizing `identity:delete_project(project_id=a80ba44232be408b8df81737a22953d2)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:54.881068 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-277ca1db-30b3-4f56-97a0-170444f5bdce tempest-VolumeQuotaClassesTest-790700512 tempest-VolumeQuotaClassesTest-790700512-project-admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:54.930038 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-277ca1db-30b3-4f56-97a0-170444f5bdce tempest-VolumeQuotaClassesTest-790700512 tempest-VolumeQuotaClassesTest-790700512-project-admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a80ba44232be408b8df81737a22953d2', 'request_id': 'req-277ca1db-30b3-4f56-97a0-170444f5bdce'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:54.937578 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-277ca1db-30b3-4f56-97a0-170444f5bdce tempest-VolumeQuotaClassesTest-790700512 tempest-VolumeQuotaClassesTest-790700512-project-admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a80ba44232be408b8df81737a22953d2', 'request_id': 'req-277ca1db-30b3-4f56-97a0-170444f5bdce'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:54.943226 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-277ca1db-30b3-4f56-97a0-170444f5bdce tempest-VolumeQuotaClassesTest-790700512 tempest-VolumeQuotaClassesTest-790700512-project-admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a80ba44232be408b8df81737a22953d2', 'request_id': 'req-277ca1db-30b3-4f56-97a0-170444f5bdce'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:54.951990 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=90,keystone:INSERT=3,keystone:DELETE=7 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:36:54.956240 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-277ca1db-30b3-4f56-97a0-170444f5bdce tempest-VolumeQuotaClassesTest-790700512 tempest-VolumeQuotaClassesTest-790700512-project-admin] The token cache is being invalidate because project a80ba44232be408b8df81737a22953d2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:36:54.956812 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-277ca1db-30b3-4f56-97a0-170444f5bdce tempest-VolumeQuotaClassesTest-790700512 tempest-VolumeQuotaClassesTest-790700512-project-admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:54.958132 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-277ca1db-30b3-4f56-97a0-170444f5bdce tempest-VolumeQuotaClassesTest-790700512 tempest-VolumeQuotaClassesTest-790700512-project-admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:54.959633 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 964/1948] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:36:54 2022] DELETE /identity/v3/projects/a80ba44232be408b8df81737a22953d2 => generated 0 bytes in 98 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:36:55.152617 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-316dbf92-c7d1-481c-a18b-faf6d5105d07 tempest-VolumeQuotaClassesTest-790700512 tempest-VolumeQuotaClassesTest-790700512-project-admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:55.155216 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-316dbf92-c7d1-481c-a18b-faf6d5105d07 tempest-VolumeQuotaClassesTest-790700512 tempest-VolumeQuotaClassesTest-790700512-project-admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:55.220125 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-316dbf92-c7d1-481c-a18b-faf6d5105d07 tempest-VolumeQuotaClassesTest-790700512 tempest-VolumeQuotaClassesTest-790700512-project-admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:55.278418 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95b26966-25e2-46e3-99ae-c5f211d21c62 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:55.279207 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95b26966-25e2-46e3-99ae-c5f211d21c62 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:55.279438 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95b26966-25e2-46e3-99ae-c5f211d21c62 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:55.279633 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95b26966-25e2-46e3-99ae-c5f211d21c62 admin admin] PATH_INFO: `/v3/users/1b9ce416d14e4d2c80ef7387a65ab17e` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:55.280826 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95b26966-25e2-46e3-99ae-c5f211d21c62 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1b9ce416d14e4d2c80ef7387a65ab17e)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:55.282755 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95b26966-25e2-46e3-99ae-c5f211d21c62 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:55.343413 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-95b26966-25e2-46e3-99ae-c5f211d21c62 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1b9ce416d14e4d2c80ef7387a65ab17e'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:55.350896 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-95b26966-25e2-46e3-99ae-c5f211d21c62 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1b9ce416d14e4d2c80ef7387a65ab17e'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:55.364307 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-95b26966-25e2-46e3-99ae-c5f211d21c62 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1b9ce416d14e4d2c80ef7387a65ab17e'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:55.365163 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-95b26966-25e2-46e3-99ae-c5f211d21c62 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1b9ce416d14e4d2c80ef7387a65ab17e'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:55.373444 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-95b26966-25e2-46e3-99ae-c5f211d21c62 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1b9ce416d14e4d2c80ef7387a65ab17e'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:55.375136 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:36:55.381714 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 985/1949] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:36:55 2022] DELETE /identity/v3/users/1b9ce416d14e4d2c80ef7387a65ab17e => generated 0 bytes in 229 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:36:55.393226 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b954525a-bd79-4a6f-b0f0-a55daf4a2630 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:55.393517 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b954525a-bd79-4a6f-b0f0-a55daf4a2630 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:55.393815 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b954525a-bd79-4a6f-b0f0-a55daf4a2630 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:55.412680 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-b954525a-bd79-4a6f-b0f0-a55daf4a2630 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:36:55.413907 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-b954525a-bd79-4a6f-b0f0-a55daf4a2630 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:55.482873 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 965/1950] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:36:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:36:55.492604 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95b26966-25e2-46e3-99ae-c5f211d21c62 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:55.494025 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-95b26966-25e2-46e3-99ae-c5f211d21c62 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:55.551661 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95b26966-25e2-46e3-99ae-c5f211d21c62 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:55.555271 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69cd6b6f-2695-43fb-b129-fa30ed4314b2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:55.555880 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69cd6b6f-2695-43fb-b129-fa30ed4314b2 service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:55.556161 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69cd6b6f-2695-43fb-b129-fa30ed4314b2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:55.556436 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69cd6b6f-2695-43fb-b129-fa30ed4314b2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:55.566145 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69cd6b6f-2695-43fb-b129-fa30ed4314b2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:55.568011 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69cd6b6f-2695-43fb-b129-fa30ed4314b2 service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:55.578426 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 986/1951] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:36:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:36:55.989696 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b954525a-bd79-4a6f-b0f0-a55daf4a2630 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:55.991765 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-b954525a-bd79-4a6f-b0f0-a55daf4a2630 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:56.017288 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b954525a-bd79-4a6f-b0f0-a55daf4a2630 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:56.023838 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-898aa1a1-f611-4175-9ca5-1de98275d7f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:56.024701 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-898aa1a1-f611-4175-9ca5-1de98275d7f7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:56.024877 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-898aa1a1-f611-4175-9ca5-1de98275d7f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:56.025466 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-898aa1a1-f611-4175-9ca5-1de98275d7f7 admin admin] PATH_INFO: `/v3/projects/07a6ee99f14c4e5aaef0619b0ad0858e` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:56.026660 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-898aa1a1-f611-4175-9ca5-1de98275d7f7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=07a6ee99f14c4e5aaef0619b0ad0858e)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:56.028760 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-898aa1a1-f611-4175-9ca5-1de98275d7f7 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:56.081838 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-898aa1a1-f611-4175-9ca5-1de98275d7f7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '07a6ee99f14c4e5aaef0619b0ad0858e', 'request_id': 'req-898aa1a1-f611-4175-9ca5-1de98275d7f7'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:56.094804 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-898aa1a1-f611-4175-9ca5-1de98275d7f7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '07a6ee99f14c4e5aaef0619b0ad0858e', 'request_id': 'req-898aa1a1-f611-4175-9ca5-1de98275d7f7'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:56.101435 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-898aa1a1-f611-4175-9ca5-1de98275d7f7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '07a6ee99f14c4e5aaef0619b0ad0858e', 'request_id': 'req-898aa1a1-f611-4175-9ca5-1de98275d7f7'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:56.113323 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-898aa1a1-f611-4175-9ca5-1de98275d7f7 admin admin] The token cache is being invalidate because project 07a6ee99f14c4e5aaef0619b0ad0858e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:36:56.113615 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-898aa1a1-f611-4175-9ca5-1de98275d7f7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:56.114314 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-898aa1a1-f611-4175-9ca5-1de98275d7f7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:56.115833 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 966/1952] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:36:55 2022] DELETE /identity/v3/projects/07a6ee99f14c4e5aaef0619b0ad0858e => generated 0 bytes in 127 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:36:56.120136 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69cd6b6f-2695-43fb-b129-fa30ed4314b2 service neutron] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:56.130530 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69cd6b6f-2695-43fb-b129-fa30ed4314b2 service neutron] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:56.190275 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb0b17b0-ff32-4c55-a5da-eedbe3876ba0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:56.190948 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb0b17b0-ff32-4c55-a5da-eedbe3876ba0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:56.191183 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb0b17b0-ff32-4c55-a5da-eedbe3876ba0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:56.191457 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb0b17b0-ff32-4c55-a5da-eedbe3876ba0 admin admin] PATH_INFO: `/v3/users/202d4ffe4d5447e5b1342711b599d884` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:56.192584 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb0b17b0-ff32-4c55-a5da-eedbe3876ba0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=202d4ffe4d5447e5b1342711b599d884)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:56.194526 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb0b17b0-ff32-4c55-a5da-eedbe3876ba0 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:56.240834 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-eb0b17b0-ff32-4c55-a5da-eedbe3876ba0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '202d4ffe4d5447e5b1342711b599d884'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:56.248793 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-eb0b17b0-ff32-4c55-a5da-eedbe3876ba0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '202d4ffe4d5447e5b1342711b599d884'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:56.260948 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-eb0b17b0-ff32-4c55-a5da-eedbe3876ba0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '202d4ffe4d5447e5b1342711b599d884'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:56.261609 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-eb0b17b0-ff32-4c55-a5da-eedbe3876ba0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '202d4ffe4d5447e5b1342711b599d884'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:56.269429 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-eb0b17b0-ff32-4c55-a5da-eedbe3876ba0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '202d4ffe4d5447e5b1342711b599d884'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:56.275421 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 987/1953] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:36:56 2022] DELETE /identity/v3/users/202d4ffe4d5447e5b1342711b599d884 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:36:56.380472 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-898aa1a1-f611-4175-9ca5-1de98275d7f7 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:56.380472 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-898aa1a1-f611-4175-9ca5-1de98275d7f7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:56.436489 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-898aa1a1-f611-4175-9ca5-1de98275d7f7 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:56.485227 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:36:56.500928 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a39b24f-7d84-4984-a86f-f84afd62d132 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ffbc136c77b84e8c87128f7247f4ec03', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:56.501687 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a39b24f-7d84-4984-a86f-f84afd62d132 service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:56.502173 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a39b24f-7d84-4984-a86f-f84afd62d132 service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:56.502862 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a39b24f-7d84-4984-a86f-f84afd62d132 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:56.505452 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-7a39b24f-7d84-4984-a86f-f84afd62d132 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:56.592947 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb0b17b0-ff32-4c55-a5da-eedbe3876ba0 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:56.594670 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-eb0b17b0-ff32-4c55-a5da-eedbe3876ba0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:56.608747 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a39b24f-7d84-4984-a86f-f84afd62d132 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:56.608747 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a39b24f-7d84-4984-a86f-f84afd62d132 service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:56.617896 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 967/1954] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:36:56 2022] GET /identity/v3/auth/tokens => generated 3319 bytes in 245 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:36:56.625656 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:36:56.677161 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb0b17b0-ff32-4c55-a5da-eedbe3876ba0 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:56.682860 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-251d27d7-6945-4f89-9a1c-608d9e4d606a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:56.683598 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-251d27d7-6945-4f89-9a1c-608d9e4d606a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:56.683883 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-251d27d7-6945-4f89-9a1c-608d9e4d606a admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:56.684735 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-251d27d7-6945-4f89-9a1c-608d9e4d606a admin admin] PATH_INFO: `/v3/projects/da8d5bbb0c394e719169742c6a7e3818` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:56.686017 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-251d27d7-6945-4f89-9a1c-608d9e4d606a admin admin] RBAC: Authorizing `identity:delete_project(project_id=da8d5bbb0c394e719169742c6a7e3818)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:56.688806 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-251d27d7-6945-4f89-9a1c-608d9e4d606a admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:56.744518 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-251d27d7-6945-4f89-9a1c-608d9e4d606a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'da8d5bbb0c394e719169742c6a7e3818', 'request_id': 'req-251d27d7-6945-4f89-9a1c-608d9e4d606a'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:56.754143 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-251d27d7-6945-4f89-9a1c-608d9e4d606a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'da8d5bbb0c394e719169742c6a7e3818', 'request_id': 'req-251d27d7-6945-4f89-9a1c-608d9e4d606a'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:56.761216 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-251d27d7-6945-4f89-9a1c-608d9e4d606a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'da8d5bbb0c394e719169742c6a7e3818', 'request_id': 'req-251d27d7-6945-4f89-9a1c-608d9e4d606a'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:56.776257 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-251d27d7-6945-4f89-9a1c-608d9e4d606a admin admin] The token cache is being invalidate because project da8d5bbb0c394e719169742c6a7e3818 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81434) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:36:56.776557 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-251d27d7-6945-4f89-9a1c-608d9e4d606a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:56.777249 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-251d27d7-6945-4f89-9a1c-608d9e4d606a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:36:56.779628 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 988/1955] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:36:56 2022] DELETE /identity/v3/projects/da8d5bbb0c394e719169742c6a7e3818 => generated 0 bytes in 188 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:36:56.784829 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a52a138-2708-449f-902a-9a906ed0e2e8 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:56.785126 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a52a138-2708-449f-902a-9a906ed0e2e8 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:56.785456 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a52a138-2708-449f-902a-9a906ed0e2e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:56.806323 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-9a52a138-2708-449f-902a-9a906ed0e2e8 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:36:56.807555 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-9a52a138-2708-449f-902a-9a906ed0e2e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:56.876566 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 968/1956] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:36:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:36:56.885080 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd8a3cbd-af71-4d0f-a57d-94ce2a02a5d0 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:56.885379 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd8a3cbd-af71-4d0f-a57d-94ce2a02a5d0 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:56.885661 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd8a3cbd-af71-4d0f-a57d-94ce2a02a5d0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:56.911738 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-bd8a3cbd-af71-4d0f-a57d-94ce2a02a5d0 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:36:56.913029 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-bd8a3cbd-af71-4d0f-a57d-94ce2a02a5d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:56.922297 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 989/1957] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:36:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:36:56.927252 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a52a138-2708-449f-902a-9a906ed0e2e8 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:56.937715 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a52a138-2708-449f-902a-9a906ed0e2e8 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:56.944876 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-208ba658-6133-4010-a763-5bbf724c2cc6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:56.946235 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-208ba658-6133-4010-a763-5bbf724c2cc6 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:56.946440 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-208ba658-6133-4010-a763-5bbf724c2cc6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:56.946726 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-208ba658-6133-4010-a763-5bbf724c2cc6 admin admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:56.947382 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-208ba658-6133-4010-a763-5bbf724c2cc6 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:36:56.947700 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-208ba658-6133-4010-a763-5bbf724c2cc6 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:56.949360 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-208ba658-6133-4010-a763-5bbf724c2cc6 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:56.967500 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 969/1958] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Sep 25 04:36:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:36:56.969820 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd8a3cbd-af71-4d0f-a57d-94ce2a02a5d0 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:56.980319 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd8a3cbd-af71-4d0f-a57d-94ce2a02a5d0 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:56.985893 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc65e6f3-9c93-431d-ba12-049e5bd42083 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:56.986590 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc65e6f3-9c93-431d-ba12-049e5bd42083 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:56.986787 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc65e6f3-9c93-431d-ba12-049e5bd42083 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:56.987748 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc65e6f3-9c93-431d-ba12-049e5bd42083 admin admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:56.988073 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc65e6f3-9c93-431d-ba12-049e5bd42083 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:56.989877 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc65e6f3-9c93-431d-ba12-049e5bd42083 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:57.028671 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 990/1959] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:36:56 2022] POST /identity/v3/projects => generated 391 bytes in 59 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:36:57.033079 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-208ba658-6133-4010-a763-5bbf724c2cc6 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:57.043201 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-208ba658-6133-4010-a763-5bbf724c2cc6 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:57.103162 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57dc44e1-da8e-48b6-ae8f-a7d11f120edd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:57.103783 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57dc44e1-da8e-48b6-ae8f-a7d11f120edd admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:57.104196 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57dc44e1-da8e-48b6-ae8f-a7d11f120edd admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:57.104482 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57dc44e1-da8e-48b6-ae8f-a7d11f120edd admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:57.105157 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57dc44e1-da8e-48b6-ae8f-a7d11f120edd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:57.107320 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57dc44e1-da8e-48b6-ae8f-a7d11f120edd admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:57.134189 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 970/1960] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:36:57 2022] POST /identity/v3/users => generated 296 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:36:57.136519 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc65e6f3-9c93-431d-ba12-049e5bd42083 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:57.147417 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc65e6f3-9c93-431d-ba12-049e5bd42083 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:57.152955 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-771fc32e-213d-4f27-823a-878ac9542e34 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:57.153568 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-771fc32e-213d-4f27-823a-878ac9542e34 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:57.153849 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-771fc32e-213d-4f27-823a-878ac9542e34 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:57.154153 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-771fc32e-213d-4f27-823a-878ac9542e34 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:57.154799 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-771fc32e-213d-4f27-823a-878ac9542e34 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:57.156765 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-771fc32e-213d-4f27-823a-878ac9542e34 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:57.167527 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 991/1961] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:36:57 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 25 04:36:57.172357 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57dc44e1-da8e-48b6-ae8f-a7d11f120edd admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:57.182640 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57dc44e1-da8e-48b6-ae8f-a7d11f120edd admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:57.187467 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee8bfe1a-8324-4bf9-831f-e44f6d5c348b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:57.188082 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee8bfe1a-8324-4bf9-831f-e44f6d5c348b admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:57.188368 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee8bfe1a-8324-4bf9-831f-e44f6d5c348b admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:57.188659 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee8bfe1a-8324-4bf9-831f-e44f6d5c348b admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:57.189331 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee8bfe1a-8324-4bf9-831f-e44f6d5c348b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:57.190944 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee8bfe1a-8324-4bf9-831f-e44f6d5c348b admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:57.192326 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:36:57.207046 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 971/1962] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:36:57 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 25 04:36:57.208785 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-771fc32e-213d-4f27-823a-878ac9542e34 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:57.223092 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-771fc32e-213d-4f27-823a-878ac9542e34 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:57.227994 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1595b6f2-2c17-45c1-9f72-93360649f8bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:57.228833 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1595b6f2-2c17-45c1-9f72-93360649f8bf admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:57.229210 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1595b6f2-2c17-45c1-9f72-93360649f8bf admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:57.229487 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1595b6f2-2c17-45c1-9f72-93360649f8bf admin admin] PATH_INFO: `/v3/projects/1f022fb733d74afda51eb37a8ba9a0e4/users/51e4259a50864792a57509c00e5d8c44/roles/8f6cc574056c4278bc9778f14dc4b411` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:57.244525 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1595b6f2-2c17-45c1-9f72-93360649f8bf admin admin] RBAC: Authorizing `identity:create_grant(project_id=1f022fb733d74afda51eb37a8ba9a0e4, user_id=51e4259a50864792a57509c00e5d8c44, role_id=8f6cc574056c4278bc9778f14dc4b411)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:57.247725 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1595b6f2-2c17-45c1-9f72-93360649f8bf admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:57.261422 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 992/1963] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:36:57 2022] PUT /identity/v3/projects/1f022fb733d74afda51eb37a8ba9a0e4/users/51e4259a50864792a57509c00e5d8c44/roles/8f6cc574056c4278bc9778f14dc4b411 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:36:57.274535 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55d3dba3-be17-4951-a773-20590c518788 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:57.274799 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55d3dba3-be17-4951-a773-20590c518788 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:57.275142 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55d3dba3-be17-4951-a773-20590c518788 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:57.308340 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-55d3dba3-be17-4951-a773-20590c518788 None None] MFA Rules not processed for user `51e4259a50864792a57509c00e5d8c44`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:36:57.310423 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-55d3dba3-be17-4951-a773-20590c518788 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:57.375616 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 972/1964] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:36:57 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:36:57.380086 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1595b6f2-2c17-45c1-9f72-93360649f8bf admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:57.431830 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:36:57.448586 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1595b6f2-2c17-45c1-9f72-93360649f8bf admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:57.454134 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-493a1f01-2841-4445-8ed2-280e154d6806 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:57.454864 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-493a1f01-2841-4445-8ed2-280e154d6806 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:57.455147 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-493a1f01-2841-4445-8ed2-280e154d6806 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:57.455414 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-493a1f01-2841-4445-8ed2-280e154d6806 admin admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:57.456052 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-493a1f01-2841-4445-8ed2-280e154d6806 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:57.458246 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-493a1f01-2841-4445-8ed2-280e154d6806 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:57.503965 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 993/1965] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:36:57 2022] POST /identity/v3/projects => generated 391 bytes in 121 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 04:36:57.505065 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55d3dba3-be17-4951-a773-20590c518788 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:57.516274 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55d3dba3-be17-4951-a773-20590c518788 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:57.573978 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1fc25a0-4b4d-422a-94ec-82f42faabb9b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:57.574664 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1fc25a0-4b4d-422a-94ec-82f42faabb9b admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:57.574991 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1fc25a0-4b4d-422a-94ec-82f42faabb9b admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:57.575375 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1fc25a0-4b4d-422a-94ec-82f42faabb9b admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:57.576028 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1fc25a0-4b4d-422a-94ec-82f42faabb9b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:57.577991 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1fc25a0-4b4d-422a-94ec-82f42faabb9b admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:57.601284 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 973/1966] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:36:57 2022] POST /identity/v3/users => generated 302 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:36:57.605149 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-493a1f01-2841-4445-8ed2-280e154d6806 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:57.612771 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1fc25a0-4b4d-422a-94ec-82f42faabb9b admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:57.614475 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-b1fc25a0-4b4d-422a-94ec-82f42faabb9b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:57.616418 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-493a1f01-2841-4445-8ed2-280e154d6806 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:57.622379 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11e5a3bc-ba51-471b-996f-ed772c018158 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:57.623055 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11e5a3bc-ba51-471b-996f-ed772c018158 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:57.623336 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11e5a3bc-ba51-471b-996f-ed772c018158 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:57.623604 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11e5a3bc-ba51-471b-996f-ed772c018158 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:57.624253 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11e5a3bc-ba51-471b-996f-ed772c018158 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:57.625946 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11e5a3bc-ba51-471b-996f-ed772c018158 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:57.637355 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 994/1967] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:36:57 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 25 04:36:57.641420 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11e5a3bc-ba51-471b-996f-ed772c018158 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:57.652008 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11e5a3bc-ba51-471b-996f-ed772c018158 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:57.658342 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7989830-37a2-47fc-b7c0-be68a1cf0edd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:57.659016 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7989830-37a2-47fc-b7c0-be68a1cf0edd admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:57.659321 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7989830-37a2-47fc-b7c0-be68a1cf0edd admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:57.659579 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7989830-37a2-47fc-b7c0-be68a1cf0edd admin admin] PATH_INFO: `/v3/projects/f0ef6d3fe7064da184d17876b9ae324b/users/f3e749fa8c6341b7869809acaa8ac611/roles/c51fe7c5ad804040879acfb54269455f` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:57.677203 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7989830-37a2-47fc-b7c0-be68a1cf0edd admin admin] RBAC: Authorizing `identity:create_grant(project_id=f0ef6d3fe7064da184d17876b9ae324b, user_id=f3e749fa8c6341b7869809acaa8ac611, role_id=c51fe7c5ad804040879acfb54269455f)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:57.679885 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7989830-37a2-47fc-b7c0-be68a1cf0edd admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:57.683924 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1fc25a0-4b4d-422a-94ec-82f42faabb9b admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:57.688875 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d3aacd0-ef3a-43c5-a907-50987f70bcb6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ffbc136c77b84e8c87128f7247f4ec03', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:57.689696 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d3aacd0-ef3a-43c5-a907-50987f70bcb6 service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:57.690125 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d3aacd0-ef3a-43c5-a907-50987f70bcb6 service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:57.690556 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d3aacd0-ef3a-43c5-a907-50987f70bcb6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:57.692375 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-6d3aacd0-ef3a-43c5-a907-50987f70bcb6 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:57.695278 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 995/1968] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:36:57 2022] PUT /identity/v3/projects/f0ef6d3fe7064da184d17876b9ae324b/users/f3e749fa8c6341b7869809acaa8ac611/roles/c51fe7c5ad804040879acfb54269455f => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 04:36:57.699088 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7989830-37a2-47fc-b7c0-be68a1cf0edd admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:57.732142 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:36:57.761861 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d3aacd0-ef3a-43c5-a907-50987f70bcb6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:57.764454 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d3aacd0-ef3a-43c5-a907-50987f70bcb6 service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:57.774605 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7989830-37a2-47fc-b7c0-be68a1cf0edd admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:57.776843 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 974/1969] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:36:57 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 164 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 25 04:36:57.781240 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-318bb466-12bc-46b6-b6f7-a2400a9a1268 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:57.781944 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-318bb466-12bc-46b6-b6f7-a2400a9a1268 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:57.782333 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-318bb466-12bc-46b6-b6f7-a2400a9a1268 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:57.782671 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-318bb466-12bc-46b6-b6f7-a2400a9a1268 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:57.783520 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-318bb466-12bc-46b6-b6f7-a2400a9a1268 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:57.785324 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-318bb466-12bc-46b6-b6f7-a2400a9a1268 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:57.797446 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 996/1970] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:36:57 2022] GET /identity/v3/roles => generated 1420 bytes in 99 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:36:57.801649 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d3aacd0-ef3a-43c5-a907-50987f70bcb6 service cinder] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:57.814008 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d3aacd0-ef3a-43c5-a907-50987f70bcb6 service cinder] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:57.819607 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9227626-5557-434a-b80e-93b4fdb5608b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:57.820274 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9227626-5557-434a-b80e-93b4fdb5608b admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:57.820556 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9227626-5557-434a-b80e-93b4fdb5608b admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:57.820872 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9227626-5557-434a-b80e-93b4fdb5608b admin admin] PATH_INFO: `/v3/projects/f0ef6d3fe7064da184d17876b9ae324b/users/f3e749fa8c6341b7869809acaa8ac611/roles/c51fe7c5ad804040879acfb54269455f` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:57.822781 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9227626-5557-434a-b80e-93b4fdb5608b admin admin] RBAC: Authorizing `identity:create_grant(project_id=f0ef6d3fe7064da184d17876b9ae324b, user_id=f3e749fa8c6341b7869809acaa8ac611, role_id=c51fe7c5ad804040879acfb54269455f)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:57.826194 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9227626-5557-434a-b80e-93b4fdb5608b admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:57.837501 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 975/1971] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:36:57 2022] PUT /identity/v3/projects/f0ef6d3fe7064da184d17876b9ae324b/users/f3e749fa8c6341b7869809acaa8ac611/roles/c51fe7c5ad804040879acfb54269455f => generated 0 bytes in 36 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:36:57.848758 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c4b08d-4731-4ea7-ac5c-af1503bbfaf5 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:57.849053 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c4b08d-4731-4ea7-ac5c-af1503bbfaf5 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:57.849328 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c4b08d-4731-4ea7-ac5c-af1503bbfaf5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:57.880457 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-65c4b08d-4731-4ea7-ac5c-af1503bbfaf5 None None] MFA Rules not processed for user `f3e749fa8c6341b7869809acaa8ac611`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:36:57.881995 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-65c4b08d-4731-4ea7-ac5c-af1503bbfaf5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:57.959354 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 997/1972] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:36:57 2022] POST /identity/v3/auth/tokens => generated 3335 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 04:36:57.970877 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9227626-5557-434a-b80e-93b4fdb5608b admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:57.972717 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-d9227626-5557-434a-b80e-93b4fdb5608b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:58.042902 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9227626-5557-434a-b80e-93b4fdb5608b admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:58.049184 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7974725b-cb4c-4ce6-bbfc-a3f52827e94e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ffbc136c77b84e8c87128f7247f4ec03', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:58.049817 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7974725b-cb4c-4ce6-bbfc-a3f52827e94e service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:58.050099 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7974725b-cb4c-4ce6-bbfc-a3f52827e94e service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:58.050371 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7974725b-cb4c-4ce6-bbfc-a3f52827e94e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:58.060337 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7974725b-cb4c-4ce6-bbfc-a3f52827e94e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:58.061821 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7974725b-cb4c-4ce6-bbfc-a3f52827e94e service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:58.072157 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 976/1973] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:36:57 2022] GET /identity/v3/auth/tokens => generated 3335 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:36:58.142342 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65c4b08d-4731-4ea7-ac5c-af1503bbfaf5 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:58.153036 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65c4b08d-4731-4ea7-ac5c-af1503bbfaf5 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:58.156967 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f8ce35a-caf1-46ba-ae86-b20a54c1bfb9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ffbc136c77b84e8c87128f7247f4ec03', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:58.157585 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f8ce35a-caf1-46ba-ae86-b20a54c1bfb9 service cinder] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:58.157904 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f8ce35a-caf1-46ba-ae86-b20a54c1bfb9 service cinder] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:58.158230 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f8ce35a-caf1-46ba-ae86-b20a54c1bfb9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:58.171915 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:36:58.219057 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f8ce35a-caf1-46ba-ae86-b20a54c1bfb9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:58.220496 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f8ce35a-caf1-46ba-ae86-b20a54c1bfb9 service cinder] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:58.229883 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 998/1974] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:36:58 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:36:59.134332 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7974725b-cb4c-4ce6-bbfc-a3f52827e94e service cinder] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:36:59.143898 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7974725b-cb4c-4ce6-bbfc-a3f52827e94e service cinder] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:36:59.147841 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-602dd574-47e9-4643-8354-dd9e0d7355c4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ffbc136c77b84e8c87128f7247f4ec03', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:36:59.148445 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-602dd574-47e9-4643-8354-dd9e0d7355c4 service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:36:59.148769 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-602dd574-47e9-4643-8354-dd9e0d7355c4 service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:36:59.149045 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-602dd574-47e9-4643-8354-dd9e0d7355c4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:36:59.150778 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-602dd574-47e9-4643-8354-dd9e0d7355c4 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:36:59.220674 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-602dd574-47e9-4643-8354-dd9e0d7355c4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:36:59.222185 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-602dd574-47e9-4643-8354-dd9e0d7355c4 service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:36:59.232202 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 977/1975] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:36:59 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:37:08.225473 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=20 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:37:09.227697 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=75,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:37:11.629715 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f8ce35a-caf1-46ba-ae86-b20a54c1bfb9 service cinder] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:11.631150 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-6f8ce35a-caf1-46ba-ae86-b20a54c1bfb9 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:37:11.704133 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f8ce35a-caf1-46ba-ae86-b20a54c1bfb9 service cinder] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:11.710074 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc0de5ee-11f1-4a17-aee0-c2400e4306b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:11.710733 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc0de5ee-11f1-4a17-aee0-c2400e4306b3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:11.711015 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc0de5ee-11f1-4a17-aee0-c2400e4306b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:11.711293 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc0de5ee-11f1-4a17-aee0-c2400e4306b3 admin admin] PATH_INFO: `/v3/users/1b22f735f34645f5b2ed7aa7b0489d48` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:11.712462 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc0de5ee-11f1-4a17-aee0-c2400e4306b3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1b22f735f34645f5b2ed7aa7b0489d48)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:11.714417 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc0de5ee-11f1-4a17-aee0-c2400e4306b3 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:11.763334 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-cc0de5ee-11f1-4a17-aee0-c2400e4306b3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1b22f735f34645f5b2ed7aa7b0489d48'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:11.770780 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-cc0de5ee-11f1-4a17-aee0-c2400e4306b3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1b22f735f34645f5b2ed7aa7b0489d48'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:11.782994 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-cc0de5ee-11f1-4a17-aee0-c2400e4306b3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1b22f735f34645f5b2ed7aa7b0489d48'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:11.783659 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-cc0de5ee-11f1-4a17-aee0-c2400e4306b3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1b22f735f34645f5b2ed7aa7b0489d48'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:11.790754 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-cc0de5ee-11f1-4a17-aee0-c2400e4306b3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1b22f735f34645f5b2ed7aa7b0489d48'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:11.796836 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 999/1976] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:37:11 2022] DELETE /identity/v3/users/1b22f735f34645f5b2ed7aa7b0489d48 => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:37:11.801323 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb9264c-f658-4f6f-9b81-d33a21c5624f None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:11.801778 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb9264c-f658-4f6f-9b81-d33a21c5624f None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:11.801868 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb9264c-f658-4f6f-9b81-d33a21c5624f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:11.821843 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-1bb9264c-f658-4f6f-9b81-d33a21c5624f None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:37:11.822975 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-1bb9264c-f658-4f6f-9b81-d33a21c5624f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:37:11.888554 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 978/1977] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:37:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 04:37:11.898280 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc0de5ee-11f1-4a17-aee0-c2400e4306b3 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:11.899698 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-cc0de5ee-11f1-4a17-aee0-c2400e4306b3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:37:11.956831 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc0de5ee-11f1-4a17-aee0-c2400e4306b3 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:11.960744 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afefb960-a9e6-4e05-b120-a6ce393b2988 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:11.961362 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afefb960-a9e6-4e05-b120-a6ce393b2988 service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:11.961630 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afefb960-a9e6-4e05-b120-a6ce393b2988 service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:11.961904 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afefb960-a9e6-4e05-b120-a6ce393b2988 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:11.970903 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afefb960-a9e6-4e05-b120-a6ce393b2988 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:11.972330 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afefb960-a9e6-4e05-b120-a6ce393b2988 service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:11.981774 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1000/1978] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:37:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:37:12.419913 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bb9264c-f658-4f6f-9b81-d33a21c5624f None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:12.421697 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-1bb9264c-f658-4f6f-9b81-d33a21c5624f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:37:12.440337 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bb9264c-f658-4f6f-9b81-d33a21c5624f None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:12.445891 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92a07715-ac28-4856-ac76-3ec5db76614e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:12.446647 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92a07715-ac28-4856-ac76-3ec5db76614e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:12.447007 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92a07715-ac28-4856-ac76-3ec5db76614e admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:12.447357 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92a07715-ac28-4856-ac76-3ec5db76614e admin admin] PATH_INFO: `/v3/projects/d76a809856fb44f68a3a7358ba8ad381` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:12.448855 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92a07715-ac28-4856-ac76-3ec5db76614e admin admin] RBAC: Authorizing `identity:delete_project(project_id=d76a809856fb44f68a3a7358ba8ad381)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:12.451402 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92a07715-ac28-4856-ac76-3ec5db76614e admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:12.519195 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-92a07715-ac28-4856-ac76-3ec5db76614e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd76a809856fb44f68a3a7358ba8ad381', 'request_id': 'req-92a07715-ac28-4856-ac76-3ec5db76614e'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:12.531115 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afefb960-a9e6-4e05-b120-a6ce393b2988 service neutron] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:12.534681 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-afefb960-a9e6-4e05-b120-a6ce393b2988 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:37:12.544345 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-92a07715-ac28-4856-ac76-3ec5db76614e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd76a809856fb44f68a3a7358ba8ad381', 'request_id': 'req-92a07715-ac28-4856-ac76-3ec5db76614e'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:12.551502 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afefb960-a9e6-4e05-b120-a6ce393b2988 service neutron] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:12.556359 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-92a07715-ac28-4856-ac76-3ec5db76614e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd76a809856fb44f68a3a7358ba8ad381', 'request_id': 'req-92a07715-ac28-4856-ac76-3ec5db76614e'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:12.577188 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-92a07715-ac28-4856-ac76-3ec5db76614e admin admin] The token cache is being invalidate because project d76a809856fb44f68a3a7358ba8ad381 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:37:12.577489 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-92a07715-ac28-4856-ac76-3ec5db76614e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:12.580310 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-92a07715-ac28-4856-ac76-3ec5db76614e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:12.589418 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 979/1979] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:37:12 2022] DELETE /identity/v3/projects/d76a809856fb44f68a3a7358ba8ad381 => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:37:12.591613 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92a07715-ac28-4856-ac76-3ec5db76614e admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:12.593368 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-92a07715-ac28-4856-ac76-3ec5db76614e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:37:12.596532 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:37:12.683177 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92dff404-3bfb-415d-b777-f54fc1cf4276 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:12.686103 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92dff404-3bfb-415d-b777-f54fc1cf4276 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:12.686388 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92dff404-3bfb-415d-b777-f54fc1cf4276 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:12.686664 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92dff404-3bfb-415d-b777-f54fc1cf4276 admin admin] PATH_INFO: `/v3/users/51e4259a50864792a57509c00e5d8c44` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:12.688343 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92dff404-3bfb-415d-b777-f54fc1cf4276 admin admin] RBAC: Authorizing `identity:delete_user(user_id=51e4259a50864792a57509c00e5d8c44)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:12.689874 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92dff404-3bfb-415d-b777-f54fc1cf4276 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:12.704534 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92a07715-ac28-4856-ac76-3ec5db76614e admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:12.710195 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccf29fe4-78c7-4f8d-873b-df24523ae34e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:12.710841 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccf29fe4-78c7-4f8d-873b-df24523ae34e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:12.711127 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccf29fe4-78c7-4f8d-873b-df24523ae34e admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:12.711397 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccf29fe4-78c7-4f8d-873b-df24523ae34e admin admin] PATH_INFO: `/v3/users/98be5bb31c954612a8eb3fe2ab8041b5` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:12.712559 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccf29fe4-78c7-4f8d-873b-df24523ae34e admin admin] RBAC: Authorizing `identity:delete_user(user_id=98be5bb31c954612a8eb3fe2ab8041b5)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:12.714565 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccf29fe4-78c7-4f8d-873b-df24523ae34e admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:12.749333 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:37:12.755728 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-92dff404-3bfb-415d-b777-f54fc1cf4276 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '51e4259a50864792a57509c00e5d8c44'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:12.766107 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-92dff404-3bfb-415d-b777-f54fc1cf4276 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '51e4259a50864792a57509c00e5d8c44'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:12.790343 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-ccf29fe4-78c7-4f8d-873b-df24523ae34e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '98be5bb31c954612a8eb3fe2ab8041b5'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:12.790518 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-92dff404-3bfb-415d-b777-f54fc1cf4276 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '51e4259a50864792a57509c00e5d8c44'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:12.790518 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-92dff404-3bfb-415d-b777-f54fc1cf4276 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '51e4259a50864792a57509c00e5d8c44'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:12.790709 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-ccf29fe4-78c7-4f8d-873b-df24523ae34e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '98be5bb31c954612a8eb3fe2ab8041b5'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:12.790796 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-92dff404-3bfb-415d-b777-f54fc1cf4276 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '51e4259a50864792a57509c00e5d8c44'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:12.796272 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1001/1980] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:37:12 2022] DELETE /identity/v3/users/51e4259a50864792a57509c00e5d8c44 => generated 0 bytes in 266 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:37:12.800518 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dc8e37f-50b7-4f79-ae5f-51465bddc0e6 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:12.800911 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dc8e37f-50b7-4f79-ae5f-51465bddc0e6 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:12.801256 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dc8e37f-50b7-4f79-ae5f-51465bddc0e6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:12.803789 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-ccf29fe4-78c7-4f8d-873b-df24523ae34e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '98be5bb31c954612a8eb3fe2ab8041b5'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:12.804565 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-ccf29fe4-78c7-4f8d-873b-df24523ae34e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '98be5bb31c954612a8eb3fe2ab8041b5'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:12.814137 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-ccf29fe4-78c7-4f8d-873b-df24523ae34e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '98be5bb31c954612a8eb3fe2ab8041b5'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:12.821933 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 980/1981] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:37:12 2022] DELETE /identity/v3/users/98be5bb31c954612a8eb3fe2ab8041b5 => generated 0 bytes in 232 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:37:12.826523 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-5dc8e37f-50b7-4f79-ae5f-51465bddc0e6 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:37:12.827890 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-5dc8e37f-50b7-4f79-ae5f-51465bddc0e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:37:12.894179 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1002/1982] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:37:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 04:37:12.905820 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccf29fe4-78c7-4f8d-873b-df24523ae34e admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:12.907321 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-ccf29fe4-78c7-4f8d-873b-df24523ae34e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:37:12.969385 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccf29fe4-78c7-4f8d-873b-df24523ae34e admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:12.975045 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcb9486b-8dfb-463c-9263-cef67e45d2d1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:12.975782 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcb9486b-8dfb-463c-9263-cef67e45d2d1 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:12.976135 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcb9486b-8dfb-463c-9263-cef67e45d2d1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:12.976486 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcb9486b-8dfb-463c-9263-cef67e45d2d1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:12.988390 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcb9486b-8dfb-463c-9263-cef67e45d2d1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:12.990064 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcb9486b-8dfb-463c-9263-cef67e45d2d1 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:13.000304 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 981/1983] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:37:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 25 04:37:13.173985 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dc8e37f-50b7-4f79-ae5f-51465bddc0e6 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:13.176506 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-5dc8e37f-50b7-4f79-ae5f-51465bddc0e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:37:13.223473 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dc8e37f-50b7-4f79-ae5f-51465bddc0e6 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:13.261007 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed1e0e6c-de58-48fc-96b2-38903badbc87 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:13.269088 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1e0e6c-de58-48fc-96b2-38903badbc87 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:13.269453 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1e0e6c-de58-48fc-96b2-38903badbc87 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:13.269742 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1e0e6c-de58-48fc-96b2-38903badbc87 admin admin] PATH_INFO: `/v3/projects/304070002d394b50b7aa0a45d9512572` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:13.272032 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed1e0e6c-de58-48fc-96b2-38903badbc87 admin admin] RBAC: Authorizing `identity:delete_project(project_id=304070002d394b50b7aa0a45d9512572)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:13.274293 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed1e0e6c-de58-48fc-96b2-38903badbc87 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:13.346889 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:37:13.374415 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-ed1e0e6c-de58-48fc-96b2-38903badbc87 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '304070002d394b50b7aa0a45d9512572', 'request_id': 'req-ed1e0e6c-de58-48fc-96b2-38903badbc87'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:13.382869 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-ed1e0e6c-de58-48fc-96b2-38903badbc87 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '304070002d394b50b7aa0a45d9512572', 'request_id': 'req-ed1e0e6c-de58-48fc-96b2-38903badbc87'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:13.391465 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-ed1e0e6c-de58-48fc-96b2-38903badbc87 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '304070002d394b50b7aa0a45d9512572', 'request_id': 'req-ed1e0e6c-de58-48fc-96b2-38903badbc87'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:13.412728 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-ed1e0e6c-de58-48fc-96b2-38903badbc87 admin admin] The token cache is being invalidate because project 304070002d394b50b7aa0a45d9512572 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81434) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:37:13.412728 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-ed1e0e6c-de58-48fc-96b2-38903badbc87 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:13.418870 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-ed1e0e6c-de58-48fc-96b2-38903badbc87 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:13.425757 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1003/1984] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:37:13 2022] DELETE /identity/v3/projects/304070002d394b50b7aa0a45d9512572 => generated 0 bytes in 253 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:37:13.429766 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcb9486b-8dfb-463c-9263-cef67e45d2d1 service neutron] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:13.445945 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcb9486b-8dfb-463c-9263-cef67e45d2d1 service neutron] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:13.451048 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed1e0e6c-de58-48fc-96b2-38903badbc87 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:13.452524 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-ed1e0e6c-de58-48fc-96b2-38903badbc87 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:37:13.564355 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41d46ad0-756a-4216-9206-41d90a62ec96 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:13.566939 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41d46ad0-756a-4216-9206-41d90a62ec96 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:13.567442 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41d46ad0-756a-4216-9206-41d90a62ec96 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:13.568019 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41d46ad0-756a-4216-9206-41d90a62ec96 admin admin] PATH_INFO: `/v3/users/b9464098dc944ab7bb55ec08742e128e` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:13.569038 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed1e0e6c-de58-48fc-96b2-38903badbc87 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:13.570734 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41d46ad0-756a-4216-9206-41d90a62ec96 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b9464098dc944ab7bb55ec08742e128e)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:13.573988 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41d46ad0-756a-4216-9206-41d90a62ec96 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:13.576269 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ad0706e-c009-44e0-b13c-c93ff223dfa6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:13.577066 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ad0706e-c009-44e0-b13c-c93ff223dfa6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:13.577419 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ad0706e-c009-44e0-b13c-c93ff223dfa6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:13.577789 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ad0706e-c009-44e0-b13c-c93ff223dfa6 admin admin] PATH_INFO: `/v3/projects/1f022fb733d74afda51eb37a8ba9a0e4` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:13.579165 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ad0706e-c009-44e0-b13c-c93ff223dfa6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1f022fb733d74afda51eb37a8ba9a0e4)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:13.581476 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ad0706e-c009-44e0-b13c-c93ff223dfa6 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:13.701669 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-8ad0706e-c009-44e0-b13c-c93ff223dfa6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1f022fb733d74afda51eb37a8ba9a0e4', 'request_id': 'req-8ad0706e-c009-44e0-b13c-c93ff223dfa6'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:13.708933 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-41d46ad0-756a-4216-9206-41d90a62ec96 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b9464098dc944ab7bb55ec08742e128e'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:13.711263 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:DELETE=13,keystone:SELECT=86,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:37:13.713972 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-8ad0706e-c009-44e0-b13c-c93ff223dfa6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1f022fb733d74afda51eb37a8ba9a0e4', 'request_id': 'req-8ad0706e-c009-44e0-b13c-c93ff223dfa6'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:13.719867 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-41d46ad0-756a-4216-9206-41d90a62ec96 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b9464098dc944ab7bb55ec08742e128e'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:13.724366 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-8ad0706e-c009-44e0-b13c-c93ff223dfa6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1f022fb733d74afda51eb37a8ba9a0e4', 'request_id': 'req-8ad0706e-c009-44e0-b13c-c93ff223dfa6'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:13.741439 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-8ad0706e-c009-44e0-b13c-c93ff223dfa6 admin admin] The token cache is being invalidate because project 1f022fb733d74afda51eb37a8ba9a0e4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81434) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:37:13.741439 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-8ad0706e-c009-44e0-b13c-c93ff223dfa6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:13.742201 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-8ad0706e-c009-44e0-b13c-c93ff223dfa6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:13.744126 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1004/1985] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:37:13 2022] DELETE /identity/v3/projects/1f022fb733d74afda51eb37a8ba9a0e4 => generated 0 bytes in 293 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:37:13.748218 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ad0706e-c009-44e0-b13c-c93ff223dfa6 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:13.749872 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-8ad0706e-c009-44e0-b13c-c93ff223dfa6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:37:13.752140 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-41d46ad0-756a-4216-9206-41d90a62ec96 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b9464098dc944ab7bb55ec08742e128e'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:13.752882 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-41d46ad0-756a-4216-9206-41d90a62ec96 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b9464098dc944ab7bb55ec08742e128e'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:13.764039 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-41d46ad0-756a-4216-9206-41d90a62ec96 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b9464098dc944ab7bb55ec08742e128e'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:13.771534 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 982/1986] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:37:13 2022] DELETE /identity/v3/users/b9464098dc944ab7bb55ec08742e128e => generated 0 bytes in 342 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 04:37:13.838318 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=87,keystone:DELETE=11,keystone:INSERT=2 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:37:13.852972 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ad0706e-c009-44e0-b13c-c93ff223dfa6 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:13.935437 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9a58bc1-d61d-4863-b925-4d724ba02484 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:13.936113 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a58bc1-d61d-4863-b925-4d724ba02484 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:13.936402 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a58bc1-d61d-4863-b925-4d724ba02484 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:13.936697 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a58bc1-d61d-4863-b925-4d724ba02484 admin admin] PATH_INFO: `/v3/users/f3e749fa8c6341b7869809acaa8ac611` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:13.937859 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9a58bc1-d61d-4863-b925-4d724ba02484 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f3e749fa8c6341b7869809acaa8ac611)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:13.939972 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9a58bc1-d61d-4863-b925-4d724ba02484 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:13.996800 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-c9a58bc1-d61d-4863-b925-4d724ba02484 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f3e749fa8c6341b7869809acaa8ac611'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:14.004733 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-c9a58bc1-d61d-4863-b925-4d724ba02484 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f3e749fa8c6341b7869809acaa8ac611'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:14.029339 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-c9a58bc1-d61d-4863-b925-4d724ba02484 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f3e749fa8c6341b7869809acaa8ac611'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:14.030533 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-c9a58bc1-d61d-4863-b925-4d724ba02484 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f3e749fa8c6341b7869809acaa8ac611'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:14.037986 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-c9a58bc1-d61d-4863-b925-4d724ba02484 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f3e749fa8c6341b7869809acaa8ac611'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:14.045117 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1005/1987] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:37:13 2022] DELETE /identity/v3/users/f3e749fa8c6341b7869809acaa8ac611 => generated 0 bytes in 297 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 04:37:14.123691 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41d46ad0-756a-4216-9206-41d90a62ec96 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:14.125522 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-41d46ad0-756a-4216-9206-41d90a62ec96 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:37:14.150319 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41d46ad0-756a-4216-9206-41d90a62ec96 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:14.229716 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a233326-cd58-4f67-a9a3-36319542d043 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:14.230425 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a233326-cd58-4f67-a9a3-36319542d043 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:14.231306 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a233326-cd58-4f67-a9a3-36319542d043 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:14.231646 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a233326-cd58-4f67-a9a3-36319542d043 admin admin] PATH_INFO: `/v3/projects/85141bb6e8944b2a8deda483fb0d4352` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:14.236399 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a233326-cd58-4f67-a9a3-36319542d043 admin admin] RBAC: Authorizing `identity:delete_project(project_id=85141bb6e8944b2a8deda483fb0d4352)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:14.238387 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a233326-cd58-4f67-a9a3-36319542d043 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:14.287336 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-3a233326-cd58-4f67-a9a3-36319542d043 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '85141bb6e8944b2a8deda483fb0d4352', 'request_id': 'req-3a233326-cd58-4f67-a9a3-36319542d043'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:14.295262 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-3a233326-cd58-4f67-a9a3-36319542d043 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '85141bb6e8944b2a8deda483fb0d4352', 'request_id': 'req-3a233326-cd58-4f67-a9a3-36319542d043'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:14.302006 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-3a233326-cd58-4f67-a9a3-36319542d043 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '85141bb6e8944b2a8deda483fb0d4352', 'request_id': 'req-3a233326-cd58-4f67-a9a3-36319542d043'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:14.314735 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-3a233326-cd58-4f67-a9a3-36319542d043 admin admin] The token cache is being invalidate because project 85141bb6e8944b2a8deda483fb0d4352 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:37:14.315030 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-3a233326-cd58-4f67-a9a3-36319542d043 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:14.315694 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-3a233326-cd58-4f67-a9a3-36319542d043 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:14.317243 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 983/1988] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:37:14 2022] DELETE /identity/v3/projects/85141bb6e8944b2a8deda483fb0d4352 => generated 0 bytes in 194 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:37:14.322469 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b75d199c-c5c4-414e-9fb8-e74d1d48fe0c None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:14.322740 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b75d199c-c5c4-414e-9fb8-e74d1d48fe0c None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:14.323054 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b75d199c-c5c4-414e-9fb8-e74d1d48fe0c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:14.344231 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a233326-cd58-4f67-a9a3-36319542d043 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:14.345626 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-b75d199c-c5c4-414e-9fb8-e74d1d48fe0c None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:37:14.345744 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-3a233326-cd58-4f67-a9a3-36319542d043 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:37:14.346837 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-b75d199c-c5c4-414e-9fb8-e74d1d48fe0c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:37:14.412179 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1006/1989] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:37:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:37:14.420322 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:INSERT=2,keystone:DELETE=10,keystone:SELECT=88 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:37:14.421460 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e902507-dc4c-4518-8842-ba64ab42af8e None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:14.421752 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e902507-dc4c-4518-8842-ba64ab42af8e None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:14.422120 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e902507-dc4c-4518-8842-ba64ab42af8e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:14.425580 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a233326-cd58-4f67-a9a3-36319542d043 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:14.434123 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f42179c2-2262-4bc8-b02a-d59c2ec6864f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:14.434783 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f42179c2-2262-4bc8-b02a-d59c2ec6864f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:14.435071 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f42179c2-2262-4bc8-b02a-d59c2ec6864f admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:14.435351 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f42179c2-2262-4bc8-b02a-d59c2ec6864f admin admin] PATH_INFO: `/v3/projects/f0ef6d3fe7064da184d17876b9ae324b` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:14.436507 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f42179c2-2262-4bc8-b02a-d59c2ec6864f admin admin] RBAC: Authorizing `identity:delete_project(project_id=f0ef6d3fe7064da184d17876b9ae324b)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:14.439163 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f42179c2-2262-4bc8-b02a-d59c2ec6864f admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:14.445118 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-6e902507-dc4c-4518-8842-ba64ab42af8e None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:37:14.446338 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-6e902507-dc4c-4518-8842-ba64ab42af8e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:37:14.455006 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1007/1990] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:37:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 04:37:14.459400 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e902507-dc4c-4518-8842-ba64ab42af8e None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:14.470231 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e902507-dc4c-4518-8842-ba64ab42af8e None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:14.475910 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-232c626d-9877-49cd-b9a0-ca2ccf65c2fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:14.476537 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-232c626d-9877-49cd-b9a0-ca2ccf65c2fb admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:14.476848 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-232c626d-9877-49cd-b9a0-ca2ccf65c2fb admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:14.477124 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-232c626d-9877-49cd-b9a0-ca2ccf65c2fb admin admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:14.477778 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-232c626d-9877-49cd-b9a0-ca2ccf65c2fb admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:37:14.478088 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-232c626d-9877-49cd-b9a0-ca2ccf65c2fb admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:14.479850 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-232c626d-9877-49cd-b9a0-ca2ccf65c2fb admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:14.495911 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1008/1991] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Sep 25 04:37:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:37:14.496139 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-f42179c2-2262-4bc8-b02a-d59c2ec6864f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f0ef6d3fe7064da184d17876b9ae324b', 'request_id': 'req-f42179c2-2262-4bc8-b02a-d59c2ec6864f'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:14.497434 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-232c626d-9877-49cd-b9a0-ca2ccf65c2fb admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:14.502957 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:37:14.503078 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-f42179c2-2262-4bc8-b02a-d59c2ec6864f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f0ef6d3fe7064da184d17876b9ae324b', 'request_id': 'req-f42179c2-2262-4bc8-b02a-d59c2ec6864f'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:14.509375 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-232c626d-9877-49cd-b9a0-ca2ccf65c2fb admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:14.509675 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-f42179c2-2262-4bc8-b02a-d59c2ec6864f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f0ef6d3fe7064da184d17876b9ae324b', 'request_id': 'req-f42179c2-2262-4bc8-b02a-d59c2ec6864f'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:14.521935 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-f42179c2-2262-4bc8-b02a-d59c2ec6864f admin admin] The token cache is being invalidate because project f0ef6d3fe7064da184d17876b9ae324b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:37:14.522223 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-f42179c2-2262-4bc8-b02a-d59c2ec6864f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:14.522889 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-f42179c2-2262-4bc8-b02a-d59c2ec6864f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:14.524391 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 984/1992] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:37:14 2022] DELETE /identity/v3/projects/f0ef6d3fe7064da184d17876b9ae324b => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 04:37:14.529237 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c20ec78-cd11-4ee8-b779-b1511b6db0b4 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:14.529624 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c20ec78-cd11-4ee8-b779-b1511b6db0b4 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:14.529906 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c20ec78-cd11-4ee8-b779-b1511b6db0b4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:14.549932 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-1c20ec78-cd11-4ee8-b779-b1511b6db0b4 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:37:14.551127 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-1c20ec78-cd11-4ee8-b779-b1511b6db0b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:37:14.567479 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7813540-cc79-4d7d-91c6-4ff5bc682410 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:14.568093 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7813540-cc79-4d7d-91c6-4ff5bc682410 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:14.568372 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7813540-cc79-4d7d-91c6-4ff5bc682410 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:14.568661 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7813540-cc79-4d7d-91c6-4ff5bc682410 admin admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:14.569304 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7813540-cc79-4d7d-91c6-4ff5bc682410 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:14.571287 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7813540-cc79-4d7d-91c6-4ff5bc682410 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:14.600266 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1009/1993] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:37:14 2022] POST /identity/v3/projects => generated 377 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 04:37:14.604357 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7813540-cc79-4d7d-91c6-4ff5bc682410 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:14.606536 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-f7813540-cc79-4d7d-91c6-4ff5bc682410 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:37:14.618188 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 985/1994] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:37:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 04:37:14.626969 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c49b9ab-46ff-4667-b197-3dd9626b5e25 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:14.627258 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c49b9ab-46ff-4667-b197-3dd9626b5e25 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:14.627536 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c49b9ab-46ff-4667-b197-3dd9626b5e25 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:14.647850 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-6c49b9ab-46ff-4667-b197-3dd9626b5e25 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:37:14.649109 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-6c49b9ab-46ff-4667-b197-3dd9626b5e25 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:37:14.674144 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7813540-cc79-4d7d-91c6-4ff5bc682410 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:14.681216 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d55273bc-5da8-40bd-9356-ce563df8c192 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:14.682086 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d55273bc-5da8-40bd-9356-ce563df8c192 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:14.682492 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d55273bc-5da8-40bd-9356-ce563df8c192 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:14.682757 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d55273bc-5da8-40bd-9356-ce563df8c192 admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:14.683397 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d55273bc-5da8-40bd-9356-ce563df8c192 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:14.686371 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d55273bc-5da8-40bd-9356-ce563df8c192 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:14.720914 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1010/1995] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:37:14 2022] POST /identity/v3/users => generated 289 bytes in 117 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 04:37:14.725637 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d55273bc-5da8-40bd-9356-ce563df8c192 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:14.744295 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d55273bc-5da8-40bd-9356-ce563df8c192 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:14.744614 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 986/1996] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:37:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 04:37:14.748808 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c49b9ab-46ff-4667-b197-3dd9626b5e25 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:14.757439 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c50a093-7cd1-4132-9f0f-aa5e2623f9d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:14.758174 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c50a093-7cd1-4132-9f0f-aa5e2623f9d7 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:14.758529 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c50a093-7cd1-4132-9f0f-aa5e2623f9d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:14.758866 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c50a093-7cd1-4132-9f0f-aa5e2623f9d7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:14.759717 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c50a093-7cd1-4132-9f0f-aa5e2623f9d7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:14.761818 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c50a093-7cd1-4132-9f0f-aa5e2623f9d7 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:14.776568 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1011/1997] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:37:14 2022] GET /identity/v3/roles => generated 1420 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:37:14.776794 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c49b9ab-46ff-4667-b197-3dd9626b5e25 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:14.778637 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c50a093-7cd1-4132-9f0f-aa5e2623f9d7 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:14.780511 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c359e901-a0ed-4b97-a646-9d90d7792e36 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:14.781204 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c359e901-a0ed-4b97-a646-9d90d7792e36 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:14.781490 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c359e901-a0ed-4b97-a646-9d90d7792e36 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:14.782789 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c359e901-a0ed-4b97-a646-9d90d7792e36 admin admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:14.783621 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c359e901-a0ed-4b97-a646-9d90d7792e36 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:37:14.783986 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c359e901-a0ed-4b97-a646-9d90d7792e36 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:14.786228 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c359e901-a0ed-4b97-a646-9d90d7792e36 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:14.791016 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c50a093-7cd1-4132-9f0f-aa5e2623f9d7 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:14.798002 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7661ebd9-6f1d-4950-b2fa-0ec9f0a28e76 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:14.798715 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7661ebd9-6f1d-4950-b2fa-0ec9f0a28e76 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:14.799067 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7661ebd9-6f1d-4950-b2fa-0ec9f0a28e76 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:14.799506 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7661ebd9-6f1d-4950-b2fa-0ec9f0a28e76 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:14.800209 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 987/1998] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Sep 25 04:37:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:37:14.800325 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7661ebd9-6f1d-4950-b2fa-0ec9f0a28e76 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:14.802298 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7661ebd9-6f1d-4950-b2fa-0ec9f0a28e76 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:14.805300 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c359e901-a0ed-4b97-a646-9d90d7792e36 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:14.817644 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1012/1999] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:37:14 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 25 04:37:14.817855 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c359e901-a0ed-4b97-a646-9d90d7792e36 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:14.821426 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7661ebd9-6f1d-4950-b2fa-0ec9f0a28e76 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:14.823226 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-652e7bef-b585-4002-8b44-002d1eb7f7cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:14.823920 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-652e7bef-b585-4002-8b44-002d1eb7f7cc admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:14.824173 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-652e7bef-b585-4002-8b44-002d1eb7f7cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:14.824463 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-652e7bef-b585-4002-8b44-002d1eb7f7cc admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:14.825150 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-652e7bef-b585-4002-8b44-002d1eb7f7cc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:14.827160 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-652e7bef-b585-4002-8b44-002d1eb7f7cc admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:14.838410 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7661ebd9-6f1d-4950-b2fa-0ec9f0a28e76 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:14.842722 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:37:14.847607 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-005925d8-4b83-4459-805a-1dbb1a254d4b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:14.849596 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-005925d8-4b83-4459-805a-1dbb1a254d4b admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:14.850127 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-005925d8-4b83-4459-805a-1dbb1a254d4b admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:14.850623 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-005925d8-4b83-4459-805a-1dbb1a254d4b admin admin] PATH_INFO: `/v3/projects/00f66d2c942b419abc8d9d010c5a57f3/users/329e3f9810b648218dd45a8b489adb1d/roles/8f6cc574056c4278bc9778f14dc4b411` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:14.861052 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 988/2000] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:37:14 2022] POST /identity/v3/projects => generated 379 bytes in 57 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:37:14.864950 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-652e7bef-b585-4002-8b44-002d1eb7f7cc admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:14.870462 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-005925d8-4b83-4459-805a-1dbb1a254d4b admin admin] RBAC: Authorizing `identity:create_grant(project_id=00f66d2c942b419abc8d9d010c5a57f3, user_id=329e3f9810b648218dd45a8b489adb1d, role_id=8f6cc574056c4278bc9778f14dc4b411)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:14.874036 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-005925d8-4b83-4459-805a-1dbb1a254d4b admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:14.897561 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1013/2001] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:37:14 2022] PUT /identity/v3/projects/00f66d2c942b419abc8d9d010c5a57f3/users/329e3f9810b648218dd45a8b489adb1d/roles/8f6cc574056c4278bc9778f14dc4b411 => generated 0 bytes in 73 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 04:37:14.900847 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-671dbf05-4298-4428-b516-a03b3375387e None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:14.901159 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-671dbf05-4298-4428-b516-a03b3375387e None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:14.901433 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-671dbf05-4298-4428-b516-a03b3375387e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:14.924896 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-671dbf05-4298-4428-b516-a03b3375387e None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:37:14.926134 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-671dbf05-4298-4428-b516-a03b3375387e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:37:14.948270 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-652e7bef-b585-4002-8b44-002d1eb7f7cc admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:14.976414 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:37:15.027197 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3658a7b4-829f-481e-ae8e-adbbd56aee22 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:15.028562 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3658a7b4-829f-481e-ae8e-adbbd56aee22 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:15.028841 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3658a7b4-829f-481e-ae8e-adbbd56aee22 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:15.029093 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3658a7b4-829f-481e-ae8e-adbbd56aee22 admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:15.029736 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3658a7b4-829f-481e-ae8e-adbbd56aee22 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:15.031671 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3658a7b4-829f-481e-ae8e-adbbd56aee22 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:15.035072 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1014/2002] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:37:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:37:15.045659 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-671dbf05-4298-4428-b516-a03b3375387e None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:15.046942 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-671dbf05-4298-4428-b516-a03b3375387e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:37:15.069516 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 989/2003] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:37:14 2022] POST /identity/v3/users => generated 290 bytes in 205 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 04:37:15.076762 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3658a7b4-829f-481e-ae8e-adbbd56aee22 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:15.087274 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3658a7b4-829f-481e-ae8e-adbbd56aee22 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:15.093282 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef9dded0-7766-495e-9cc7-49af1f6db1fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:15.093956 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef9dded0-7766-495e-9cc7-49af1f6db1fe admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:15.094242 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef9dded0-7766-495e-9cc7-49af1f6db1fe admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:15.094524 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef9dded0-7766-495e-9cc7-49af1f6db1fe admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:15.095169 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef9dded0-7766-495e-9cc7-49af1f6db1fe admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:15.096867 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef9dded0-7766-495e-9cc7-49af1f6db1fe admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:15.108257 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 990/2004] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:37:15 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:37:15.112451 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef9dded0-7766-495e-9cc7-49af1f6db1fe admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:15.124379 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef9dded0-7766-495e-9cc7-49af1f6db1fe admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:15.125040 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-671dbf05-4298-4428-b516-a03b3375387e None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:15.128048 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cde94eca-e7f7-490f-9d90-f015385b78ef service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:15.128814 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cde94eca-e7f7-490f-9d90-f015385b78ef service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:15.128999 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8968b4c2-e853-47f2-813e-8787b241b40c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:15.129296 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cde94eca-e7f7-490f-9d90-f015385b78ef service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:15.129627 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8968b4c2-e853-47f2-813e-8787b241b40c admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:15.129737 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cde94eca-e7f7-490f-9d90-f015385b78ef service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:15.129907 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8968b4c2-e853-47f2-813e-8787b241b40c admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:15.130190 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8968b4c2-e853-47f2-813e-8787b241b40c admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:15.130890 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8968b4c2-e853-47f2-813e-8787b241b40c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:15.132533 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8968b4c2-e853-47f2-813e-8787b241b40c admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:15.140617 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cde94eca-e7f7-490f-9d90-f015385b78ef service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:15.142144 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cde94eca-e7f7-490f-9d90-f015385b78ef service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:15.147211 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 991/2005] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:37:15 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 25 04:37:15.151461 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8968b4c2-e853-47f2-813e-8787b241b40c admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:15.162144 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1015/2006] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:37:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 117 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:37:15.162549 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8968b4c2-e853-47f2-813e-8787b241b40c admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:15.175210 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf614fb6-d9a0-452e-8abf-cdc8671517fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:15.176231 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf614fb6-d9a0-452e-8abf-cdc8671517fe admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:15.176769 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf614fb6-d9a0-452e-8abf-cdc8671517fe admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:15.177048 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf614fb6-d9a0-452e-8abf-cdc8671517fe admin admin] PATH_INFO: `/v3/projects/cd4c4087692c487f96d924b697158889/users/3d1602dfacfa42b9afae05009ca03f46/roles/8f6cc574056c4278bc9778f14dc4b411` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:15.201575 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf614fb6-d9a0-452e-8abf-cdc8671517fe admin admin] RBAC: Authorizing `identity:create_grant(project_id=cd4c4087692c487f96d924b697158889, user_id=3d1602dfacfa42b9afae05009ca03f46, role_id=8f6cc574056c4278bc9778f14dc4b411)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:15.204039 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf614fb6-d9a0-452e-8abf-cdc8671517fe admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:15.228799 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 992/2007] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:37:15 2022] PUT /identity/v3/projects/cd4c4087692c487f96d924b697158889/users/3d1602dfacfa42b9afae05009ca03f46/roles/8f6cc574056c4278bc9778f14dc4b411 => generated 0 bytes in 69 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 04:37:15.230564 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4b3165c-3fb3-42f2-b561-6017dce38b54 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:15.230739 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4b3165c-3fb3-42f2-b561-6017dce38b54 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:15.231015 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4b3165c-3fb3-42f2-b561-6017dce38b54 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:15.265519 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-f4b3165c-3fb3-42f2-b561-6017dce38b54 None None] MFA Rules not processed for user `3d1602dfacfa42b9afae05009ca03f46`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:37:15.266767 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-f4b3165c-3fb3-42f2-b561-6017dce38b54 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:37:15.339501 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1016/2008] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:37:15 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:37:15.344202 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf614fb6-d9a0-452e-8abf-cdc8671517fe admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:15.393562 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:37:15.413984 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf614fb6-d9a0-452e-8abf-cdc8671517fe admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:15.420546 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97d6d49c-42b8-495c-903b-d13d1d4ef0f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:15.421270 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97d6d49c-42b8-495c-903b-d13d1d4ef0f3 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:15.421627 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97d6d49c-42b8-495c-903b-d13d1d4ef0f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:15.421908 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97d6d49c-42b8-495c-903b-d13d1d4ef0f3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:15.422562 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97d6d49c-42b8-495c-903b-d13d1d4ef0f3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:15.424529 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97d6d49c-42b8-495c-903b-d13d1d4ef0f3 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:15.454961 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 993/2009] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:37:15 2022] POST /identity/v3/projects => generated 377 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 04:37:15.456092 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4b3165c-3fb3-42f2-b561-6017dce38b54 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:15.471338 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4b3165c-3fb3-42f2-b561-6017dce38b54 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:15.538008 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c57e944-188d-4f9a-b64e-535764016ab4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:15.538718 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c57e944-188d-4f9a-b64e-535764016ab4 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:15.539080 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c57e944-188d-4f9a-b64e-535764016ab4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:15.539451 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c57e944-188d-4f9a-b64e-535764016ab4 admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:15.540178 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c57e944-188d-4f9a-b64e-535764016ab4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:15.542556 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c57e944-188d-4f9a-b64e-535764016ab4 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:15.554374 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:37:15.576724 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1017/2010] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:37:15 2022] POST /identity/v3/users => generated 295 bytes in 120 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 04:37:15.579620 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97d6d49c-42b8-495c-903b-d13d1d4ef0f3 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:15.590969 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97d6d49c-42b8-495c-903b-d13d1d4ef0f3 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:15.596248 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8560fc5b-f2b5-49d6-9c77-4e1e9aa6f01e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:15.596923 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8560fc5b-f2b5-49d6-9c77-4e1e9aa6f01e admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:15.597256 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8560fc5b-f2b5-49d6-9c77-4e1e9aa6f01e admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:15.597531 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8560fc5b-f2b5-49d6-9c77-4e1e9aa6f01e admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:15.598164 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8560fc5b-f2b5-49d6-9c77-4e1e9aa6f01e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:15.599819 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8560fc5b-f2b5-49d6-9c77-4e1e9aa6f01e admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:15.612281 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 994/2011] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:37:15 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:37:15.616682 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c57e944-188d-4f9a-b64e-535764016ab4 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:15.628086 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c57e944-188d-4f9a-b64e-535764016ab4 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:15.632895 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da8cc937-696b-4e76-91dd-25627f6781e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:15.633583 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da8cc937-696b-4e76-91dd-25627f6781e3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:15.633869 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da8cc937-696b-4e76-91dd-25627f6781e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:15.634144 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da8cc937-696b-4e76-91dd-25627f6781e3 admin admin] PATH_INFO: `/v3/projects/756d9ebea37b4dcc91ef984de764dde1/users/d380d81d908b46d5a52fd97787ef724e/roles/c51fe7c5ad804040879acfb54269455f` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:15.653294 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da8cc937-696b-4e76-91dd-25627f6781e3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=756d9ebea37b4dcc91ef984de764dde1, user_id=d380d81d908b46d5a52fd97787ef724e, role_id=c51fe7c5ad804040879acfb54269455f)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:15.655932 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da8cc937-696b-4e76-91dd-25627f6781e3 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:15.670909 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1018/2012] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:37:15 2022] PUT /identity/v3/projects/756d9ebea37b4dcc91ef984de764dde1/users/d380d81d908b46d5a52fd97787ef724e/roles/c51fe7c5ad804040879acfb54269455f => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:37:15.674972 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8560fc5b-f2b5-49d6-9c77-4e1e9aa6f01e admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:15.686105 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8560fc5b-f2b5-49d6-9c77-4e1e9aa6f01e admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:15.748807 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8311ddd-a663-43af-a7a7-bf4fab6df257 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:15.749786 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8311ddd-a663-43af-a7a7-bf4fab6df257 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:15.749901 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8311ddd-a663-43af-a7a7-bf4fab6df257 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:15.750094 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8311ddd-a663-43af-a7a7-bf4fab6df257 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:15.750746 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8311ddd-a663-43af-a7a7-bf4fab6df257 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:15.752402 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8311ddd-a663-43af-a7a7-bf4fab6df257 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:15.767573 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 995/2013] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:37:15 2022] GET /identity/v3/roles => generated 1420 bytes in 90 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:37:15.768848 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da8cc937-696b-4e76-91dd-25627f6781e3 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:15.779811 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da8cc937-696b-4e76-91dd-25627f6781e3 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:15.785505 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffc1394a-f265-4100-93c4-430547a251ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:15.786201 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc1394a-f265-4100-93c4-430547a251ec admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:15.786479 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc1394a-f265-4100-93c4-430547a251ec admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:15.786754 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc1394a-f265-4100-93c4-430547a251ec admin admin] PATH_INFO: `/v3/projects/756d9ebea37b4dcc91ef984de764dde1/users/d380d81d908b46d5a52fd97787ef724e/roles/c51fe7c5ad804040879acfb54269455f` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:15.788779 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffc1394a-f265-4100-93c4-430547a251ec admin admin] RBAC: Authorizing `identity:create_grant(project_id=756d9ebea37b4dcc91ef984de764dde1, user_id=d380d81d908b46d5a52fd97787ef724e, role_id=c51fe7c5ad804040879acfb54269455f)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:15.791402 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffc1394a-f265-4100-93c4-430547a251ec admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:15.804578 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1019/2014] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:37:15 2022] PUT /identity/v3/projects/756d9ebea37b4dcc91ef984de764dde1/users/d380d81d908b46d5a52fd97787ef724e/roles/c51fe7c5ad804040879acfb54269455f => generated 0 bytes in 36 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 04:37:15.815614 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e0c9210-0755-40b4-8412-444307ccbf57 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:15.815910 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e0c9210-0755-40b4-8412-444307ccbf57 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:15.816195 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e0c9210-0755-40b4-8412-444307ccbf57 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:15.865483 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-7e0c9210-0755-40b4-8412-444307ccbf57 None None] MFA Rules not processed for user `d380d81d908b46d5a52fd97787ef724e`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:37:15.866741 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-7e0c9210-0755-40b4-8412-444307ccbf57 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:37:15.951758 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 996/2015] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:37:15 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:37:15.963501 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffc1394a-f265-4100-93c4-430547a251ec admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:15.966117 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-ffc1394a-f265-4100-93c4-430547a251ec admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:37:16.041507 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffc1394a-f265-4100-93c4-430547a251ec admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:16.046207 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23f80818-6d67-4a5a-bcba-e4606de68c3b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ffbc136c77b84e8c87128f7247f4ec03', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:16.046822 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23f80818-6d67-4a5a-bcba-e4606de68c3b service cinder] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:16.047100 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23f80818-6d67-4a5a-bcba-e4606de68c3b service cinder] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:16.047371 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23f80818-6d67-4a5a-bcba-e4606de68c3b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:16.057311 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23f80818-6d67-4a5a-bcba-e4606de68c3b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:16.058895 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23f80818-6d67-4a5a-bcba-e4606de68c3b service cinder] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:16.069932 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1020/2016] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:37:15 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:37:16.723651 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e0c9210-0755-40b4-8412-444307ccbf57 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:16.783745 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:37:16.828500 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e0c9210-0755-40b4-8412-444307ccbf57 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:16.836447 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fc93168-d181-4717-b753-19691e8ecb19 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:16.837438 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc93168-d181-4717-b753-19691e8ecb19 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:16.837939 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc93168-d181-4717-b753-19691e8ecb19 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:16.838174 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc93168-d181-4717-b753-19691e8ecb19 admin admin] PATH_INFO: `/v3/users/3d1602dfacfa42b9afae05009ca03f46` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:16.839897 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fc93168-d181-4717-b753-19691e8ecb19 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3d1602dfacfa42b9afae05009ca03f46)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:16.841869 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fc93168-d181-4717-b753-19691e8ecb19 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:16.899981 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-5fc93168-d181-4717-b753-19691e8ecb19 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3d1602dfacfa42b9afae05009ca03f46'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:16.914389 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-5fc93168-d181-4717-b753-19691e8ecb19 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3d1602dfacfa42b9afae05009ca03f46'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:16.929273 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-5fc93168-d181-4717-b753-19691e8ecb19 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3d1602dfacfa42b9afae05009ca03f46'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:16.930184 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-5fc93168-d181-4717-b753-19691e8ecb19 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3d1602dfacfa42b9afae05009ca03f46'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:16.938922 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-5fc93168-d181-4717-b753-19691e8ecb19 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3d1602dfacfa42b9afae05009ca03f46'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:16.946785 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 997/2017] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:37:16 2022] DELETE /identity/v3/users/3d1602dfacfa42b9afae05009ca03f46 => generated 0 bytes in 223 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:37:16.951288 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c760b6fa-d547-4970-b685-116825606127 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:16.952003 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c760b6fa-d547-4970-b685-116825606127 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:16.952320 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c760b6fa-d547-4970-b685-116825606127 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:16.975735 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-c760b6fa-d547-4970-b685-116825606127 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:37:16.977229 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-c760b6fa-d547-4970-b685-116825606127 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:37:17.047498 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1021/2018] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:37:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:37:17.057723 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fc93168-d181-4717-b753-19691e8ecb19 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:17.059724 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-5fc93168-d181-4717-b753-19691e8ecb19 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:37:17.145155 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fc93168-d181-4717-b753-19691e8ecb19 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:17.150477 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40109930-361e-4081-8b47-de63487791fb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:17.151392 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40109930-361e-4081-8b47-de63487791fb service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:17.151832 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40109930-361e-4081-8b47-de63487791fb service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:17.152852 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40109930-361e-4081-8b47-de63487791fb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:17.164122 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40109930-361e-4081-8b47-de63487791fb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:17.166248 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40109930-361e-4081-8b47-de63487791fb service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:17.178494 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 998/2019] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:37:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 121 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:37:17.724544 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c760b6fa-d547-4970-b685-116825606127 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:17.726159 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-c760b6fa-d547-4970-b685-116825606127 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:37:17.740031 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c760b6fa-d547-4970-b685-116825606127 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:17.745279 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13592cee-159e-4643-89fd-1c464ee50d11 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:17.746172 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13592cee-159e-4643-89fd-1c464ee50d11 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:17.746457 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13592cee-159e-4643-89fd-1c464ee50d11 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:17.746730 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13592cee-159e-4643-89fd-1c464ee50d11 admin admin] PATH_INFO: `/v3/projects/cd4c4087692c487f96d924b697158889` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:17.748092 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13592cee-159e-4643-89fd-1c464ee50d11 admin admin] RBAC: Authorizing `identity:delete_project(project_id=cd4c4087692c487f96d924b697158889)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:17.750163 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13592cee-159e-4643-89fd-1c464ee50d11 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:17.845879 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-13592cee-159e-4643-89fd-1c464ee50d11 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cd4c4087692c487f96d924b697158889', 'request_id': 'req-13592cee-159e-4643-89fd-1c464ee50d11'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:17.862216 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:INSERT=4,keystone:SELECT=91,keystone:DELETE=5 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:37:17.869895 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-13592cee-159e-4643-89fd-1c464ee50d11 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cd4c4087692c487f96d924b697158889', 'request_id': 'req-13592cee-159e-4643-89fd-1c464ee50d11'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:17.886487 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-13592cee-159e-4643-89fd-1c464ee50d11 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cd4c4087692c487f96d924b697158889', 'request_id': 'req-13592cee-159e-4643-89fd-1c464ee50d11'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:17.906848 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-13592cee-159e-4643-89fd-1c464ee50d11 admin admin] The token cache is being invalidate because project cd4c4087692c487f96d924b697158889 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81434) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:37:17.907435 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-13592cee-159e-4643-89fd-1c464ee50d11 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:17.908345 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-13592cee-159e-4643-89fd-1c464ee50d11 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:17.912366 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1022/2020] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:37:17 2022] DELETE /identity/v3/projects/cd4c4087692c487f96d924b697158889 => generated 0 bytes in 188 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:37:17.916351 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40109930-361e-4081-8b47-de63487791fb service neutron] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:17.937921 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40109930-361e-4081-8b47-de63487791fb service neutron] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:18.034509 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a7a3d46-1e75-42c6-bdb4-e7bd6b03cb2d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:18.035429 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a7a3d46-1e75-42c6-bdb4-e7bd6b03cb2d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:18.036085 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a7a3d46-1e75-42c6-bdb4-e7bd6b03cb2d admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:18.036662 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a7a3d46-1e75-42c6-bdb4-e7bd6b03cb2d admin admin] PATH_INFO: `/v3/users/d380d81d908b46d5a52fd97787ef724e` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:18.038036 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a7a3d46-1e75-42c6-bdb4-e7bd6b03cb2d admin admin] RBAC: Authorizing `identity:delete_user(user_id=d380d81d908b46d5a52fd97787ef724e)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:18.041772 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a7a3d46-1e75-42c6-bdb4-e7bd6b03cb2d admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:18.044282 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:37:18.118319 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-4a7a3d46-1e75-42c6-bdb4-e7bd6b03cb2d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd380d81d908b46d5a52fd97787ef724e'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:18.132531 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-4a7a3d46-1e75-42c6-bdb4-e7bd6b03cb2d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd380d81d908b46d5a52fd97787ef724e'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:18.147766 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-4a7a3d46-1e75-42c6-bdb4-e7bd6b03cb2d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd380d81d908b46d5a52fd97787ef724e'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:18.148687 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-4a7a3d46-1e75-42c6-bdb4-e7bd6b03cb2d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd380d81d908b46d5a52fd97787ef724e'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:18.156603 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-4a7a3d46-1e75-42c6-bdb4-e7bd6b03cb2d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd380d81d908b46d5a52fd97787ef724e'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:18.164354 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 999/2021] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:37:17 2022] DELETE /identity/v3/users/d380d81d908b46d5a52fd97787ef724e => generated 0 bytes in 249 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:37:18.539215 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13592cee-159e-4643-89fd-1c464ee50d11 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:18.541070 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-13592cee-159e-4643-89fd-1c464ee50d11 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:37:18.566130 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13592cee-159e-4643-89fd-1c464ee50d11 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:18.668874 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e5a2515-ebc3-4b27-a329-9247630d9364 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:18.669639 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e5a2515-ebc3-4b27-a329-9247630d9364 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:18.669994 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e5a2515-ebc3-4b27-a329-9247630d9364 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:18.671945 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e5a2515-ebc3-4b27-a329-9247630d9364 admin admin] PATH_INFO: `/v3/projects/756d9ebea37b4dcc91ef984de764dde1` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:18.673552 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e5a2515-ebc3-4b27-a329-9247630d9364 admin admin] RBAC: Authorizing `identity:delete_project(project_id=756d9ebea37b4dcc91ef984de764dde1)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:18.678037 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e5a2515-ebc3-4b27-a329-9247630d9364 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:18.803937 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-6e5a2515-ebc3-4b27-a329-9247630d9364 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '756d9ebea37b4dcc91ef984de764dde1', 'request_id': 'req-6e5a2515-ebc3-4b27-a329-9247630d9364'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:18.814813 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-6e5a2515-ebc3-4b27-a329-9247630d9364 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '756d9ebea37b4dcc91ef984de764dde1', 'request_id': 'req-6e5a2515-ebc3-4b27-a329-9247630d9364'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:18.828571 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-6e5a2515-ebc3-4b27-a329-9247630d9364 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '756d9ebea37b4dcc91ef984de764dde1', 'request_id': 'req-6e5a2515-ebc3-4b27-a329-9247630d9364'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:18.872049 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-6e5a2515-ebc3-4b27-a329-9247630d9364 admin admin] The token cache is being invalidate because project 756d9ebea37b4dcc91ef984de764dde1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81434) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:37:18.872214 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-6e5a2515-ebc3-4b27-a329-9247630d9364 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:18.878257 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-6e5a2515-ebc3-4b27-a329-9247630d9364 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:18.883401 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1023/2022] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:37:18 2022] DELETE /identity/v3/projects/756d9ebea37b4dcc91ef984de764dde1 => generated 0 bytes in 345 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:37:18.888894 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da926b2-eea6-4b94-9999-c6de28d8f38a None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:18.889184 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da926b2-eea6-4b94-9999-c6de28d8f38a None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:18.889461 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da926b2-eea6-4b94-9999-c6de28d8f38a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:18.927454 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-5da926b2-eea6-4b94-9999-c6de28d8f38a None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:37:18.930665 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-5da926b2-eea6-4b94-9999-c6de28d8f38a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:37:19.025958 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1000/2023] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:37:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:37:19.034241 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf1116b2-1755-437d-93a5-12644a4785fe None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:19.034578 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf1116b2-1755-437d-93a5-12644a4785fe None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:19.034904 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf1116b2-1755-437d-93a5-12644a4785fe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:19.084946 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-cf1116b2-1755-437d-93a5-12644a4785fe None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:37:19.096665 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-cf1116b2-1755-437d-93a5-12644a4785fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:37:19.109119 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1024/2024] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:37:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 76 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:37:19.113513 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5da926b2-eea6-4b94-9999-c6de28d8f38a None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:19.129943 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5da926b2-eea6-4b94-9999-c6de28d8f38a None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:19.146361 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2bb727f-3332-4d08-8b59-8dced2320f44 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:19.147001 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2bb727f-3332-4d08-8b59-8dced2320f44 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:19.147286 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2bb727f-3332-4d08-8b59-8dced2320f44 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:19.147600 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2bb727f-3332-4d08-8b59-8dced2320f44 admin admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:19.148214 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2bb727f-3332-4d08-8b59-8dced2320f44 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:37:19.148696 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2bb727f-3332-4d08-8b59-8dced2320f44 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:19.162559 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2bb727f-3332-4d08-8b59-8dced2320f44 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:19.191951 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf1116b2-1755-437d-93a5-12644a4785fe None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:19.198270 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1001/2025] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Sep 25 04:37:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:37:19.201904 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf1116b2-1755-437d-93a5-12644a4785fe None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:19.213106 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a08237e-de96-44c0-9965-166af9dcdca6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:19.213738 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a08237e-de96-44c0-9965-166af9dcdca6 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:19.214015 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a08237e-de96-44c0-9965-166af9dcdca6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:19.214286 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a08237e-de96-44c0-9965-166af9dcdca6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:19.216030 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a08237e-de96-44c0-9965-166af9dcdca6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:19.218788 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a08237e-de96-44c0-9965-166af9dcdca6 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:19.257634 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1025/2026] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:37:19 2022] POST /identity/v3/projects => generated 375 bytes in 66 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 04:37:19.261067 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2bb727f-3332-4d08-8b59-8dced2320f44 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:19.275800 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2bb727f-3332-4d08-8b59-8dced2320f44 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:19.389562 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c03119a8-873d-46b9-be78-eb3e14d76685 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:19.390196 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c03119a8-873d-46b9-be78-eb3e14d76685 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:19.390486 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c03119a8-873d-46b9-be78-eb3e14d76685 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:19.390769 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c03119a8-873d-46b9-be78-eb3e14d76685 admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:19.391420 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c03119a8-873d-46b9-be78-eb3e14d76685 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:19.393970 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c03119a8-873d-46b9-be78-eb3e14d76685 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:19.419474 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=86,keystone:DELETE=10,keystone:INSERT=4 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:37:19.434474 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1002/2027] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:37:19 2022] POST /identity/v3/users => generated 288 bytes in 174 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:37:19.437469 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a08237e-de96-44c0-9965-166af9dcdca6 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:19.448198 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a08237e-de96-44c0-9965-166af9dcdca6 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:19.455046 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e4a8435-6a4b-4732-868b-af216328ef8e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:19.455679 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e4a8435-6a4b-4732-868b-af216328ef8e admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:19.455962 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e4a8435-6a4b-4732-868b-af216328ef8e admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:19.456233 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e4a8435-6a4b-4732-868b-af216328ef8e admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:19.456960 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e4a8435-6a4b-4732-868b-af216328ef8e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:19.458905 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e4a8435-6a4b-4732-868b-af216328ef8e admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:19.470268 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1026/2028] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:37:19 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:37:19.474964 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c03119a8-873d-46b9-be78-eb3e14d76685 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:19.486025 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c03119a8-873d-46b9-be78-eb3e14d76685 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:19.492169 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74c41c1c-421d-47d5-a685-58df68997987 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:19.494186 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c41c1c-421d-47d5-a685-58df68997987 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:19.494186 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c41c1c-421d-47d5-a685-58df68997987 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:19.494186 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c41c1c-421d-47d5-a685-58df68997987 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:19.494627 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74c41c1c-421d-47d5-a685-58df68997987 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:19.498839 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74c41c1c-421d-47d5-a685-58df68997987 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:19.516913 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1003/2029] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:37:19 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:37:19.518859 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e4a8435-6a4b-4732-868b-af216328ef8e admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:19.533991 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e4a8435-6a4b-4732-868b-af216328ef8e admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:19.542355 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34200ee3-8929-4957-b6d4-7588002a0aad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:19.545696 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34200ee3-8929-4957-b6d4-7588002a0aad admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:19.546439 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34200ee3-8929-4957-b6d4-7588002a0aad admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:19.547063 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34200ee3-8929-4957-b6d4-7588002a0aad admin admin] PATH_INFO: `/v3/projects/901177d75ba64ca49037a6285e8c7b6a/users/51c1b359a50c4d87a79adc009204730d/roles/8f6cc574056c4278bc9778f14dc4b411` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:19.567615 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34200ee3-8929-4957-b6d4-7588002a0aad admin admin] RBAC: Authorizing `identity:create_grant(project_id=901177d75ba64ca49037a6285e8c7b6a, user_id=51c1b359a50c4d87a79adc009204730d, role_id=8f6cc574056c4278bc9778f14dc4b411)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:19.571257 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34200ee3-8929-4957-b6d4-7588002a0aad admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:19.612844 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1027/2030] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:37:19 2022] PUT /identity/v3/projects/901177d75ba64ca49037a6285e8c7b6a/users/51c1b359a50c4d87a79adc009204730d/roles/8f6cc574056c4278bc9778f14dc4b411 => generated 0 bytes in 81 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:37:19.616317 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-160d2ea9-d347-4775-9266-8e31ee5a7535 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:19.616666 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-160d2ea9-d347-4775-9266-8e31ee5a7535 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:19.617000 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-160d2ea9-d347-4775-9266-8e31ee5a7535 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:19.656573 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-160d2ea9-d347-4775-9266-8e31ee5a7535 None None] MFA Rules not processed for user `51c1b359a50c4d87a79adc009204730d`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:37:19.658582 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-160d2ea9-d347-4775-9266-8e31ee5a7535 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:37:19.761500 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1004/2031] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:37:19 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:37:19.772910 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34200ee3-8929-4957-b6d4-7588002a0aad admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:19.779524 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-34200ee3-8929-4957-b6d4-7588002a0aad admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:37:19.814097 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=88,keystone:DELETE=9,keystone:INSERT=3 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:37:19.894553 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34200ee3-8929-4957-b6d4-7588002a0aad admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:19.898800 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb281a42-8e07-4a2a-bbfe-55f51ce787a7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ffbc136c77b84e8c87128f7247f4ec03', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:19.899507 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb281a42-8e07-4a2a-bbfe-55f51ce787a7 service cinder] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:19.899914 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb281a42-8e07-4a2a-bbfe-55f51ce787a7 service cinder] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:19.900258 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb281a42-8e07-4a2a-bbfe-55f51ce787a7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:19.912269 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb281a42-8e07-4a2a-bbfe-55f51ce787a7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:19.914446 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb281a42-8e07-4a2a-bbfe-55f51ce787a7 service cinder] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:19.927745 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1028/2032] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:37:19 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 155 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:37:21.168967 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef0ba2ba-596f-415f-ba02-a3ebc4fa43bd None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:21.170388 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef0ba2ba-596f-415f-ba02-a3ebc4fa43bd None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:21.170716 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef0ba2ba-596f-415f-ba02-a3ebc4fa43bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:21.213633 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-ef0ba2ba-596f-415f-ba02-a3ebc4fa43bd None None] MFA Rules not processed for user `329e3f9810b648218dd45a8b489adb1d`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:37:21.215080 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-ef0ba2ba-596f-415f-ba02-a3ebc4fa43bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:37:21.310733 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1005/2033] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:37:21 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:37:21.316267 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb281a42-8e07-4a2a-bbfe-55f51ce787a7 service cinder] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:21.319108 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-cb281a42-8e07-4a2a-bbfe-55f51ce787a7 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:37:21.477443 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb281a42-8e07-4a2a-bbfe-55f51ce787a7 service cinder] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:21.487948 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdfc48df-ac36-4597-a618-1f99f873e223 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:21.488750 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdfc48df-ac36-4597-a618-1f99f873e223 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:21.489017 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdfc48df-ac36-4597-a618-1f99f873e223 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:21.489291 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdfc48df-ac36-4597-a618-1f99f873e223 admin admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:21.490130 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdfc48df-ac36-4597-a618-1f99f873e223 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:21.493317 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdfc48df-ac36-4597-a618-1f99f873e223 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:21.533658 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1029/2034] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:37:21 2022] POST /identity/v3/projects => generated 379 bytes in 219 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 04:37:21.539095 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef0ba2ba-596f-415f-ba02-a3ebc4fa43bd None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:21.555744 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef0ba2ba-596f-415f-ba02-a3ebc4fa43bd None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:21.635921 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f846749d-85e7-4739-8ab8-cfb7684be670 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:21.637528 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f846749d-85e7-4739-8ab8-cfb7684be670 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:21.637976 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f846749d-85e7-4739-8ab8-cfb7684be670 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:21.640872 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f846749d-85e7-4739-8ab8-cfb7684be670 admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:21.642216 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f846749d-85e7-4739-8ab8-cfb7684be670 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:21.645167 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f846749d-85e7-4739-8ab8-cfb7684be670 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:21.658952 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:37:21.681713 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1006/2035] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:37:21 2022] POST /identity/v3/users => generated 296 bytes in 143 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:37:21.686718 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdfc48df-ac36-4597-a618-1f99f873e223 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:21.758022 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdfc48df-ac36-4597-a618-1f99f873e223 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:21.766095 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81143aa6-bc48-4e79-a255-6861b4925e43 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:21.766857 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81143aa6-bc48-4e79-a255-6861b4925e43 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:21.767220 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81143aa6-bc48-4e79-a255-6861b4925e43 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:21.767416 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81143aa6-bc48-4e79-a255-6861b4925e43 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:21.768128 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81143aa6-bc48-4e79-a255-6861b4925e43 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:21.772008 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81143aa6-bc48-4e79-a255-6861b4925e43 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:21.794374 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1030/2036] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:37:21 2022] GET /identity/v3/roles => generated 1420 bytes in 109 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:37:21.798812 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f846749d-85e7-4739-8ab8-cfb7684be670 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:21.824631 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f846749d-85e7-4739-8ab8-cfb7684be670 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:21.832293 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf74578f-1a1f-45d1-8c3f-1066632f4f16 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:21.834794 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf74578f-1a1f-45d1-8c3f-1066632f4f16 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:21.834794 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf74578f-1a1f-45d1-8c3f-1066632f4f16 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:21.834794 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf74578f-1a1f-45d1-8c3f-1066632f4f16 admin admin] PATH_INFO: `/v3/projects/b06e471023194d10920aade4346c3bc3/users/6036528f63e3402283c4f3f96adf5f6e/roles/c51fe7c5ad804040879acfb54269455f` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:21.858711 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf74578f-1a1f-45d1-8c3f-1066632f4f16 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b06e471023194d10920aade4346c3bc3, user_id=6036528f63e3402283c4f3f96adf5f6e, role_id=c51fe7c5ad804040879acfb54269455f)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:21.862216 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf74578f-1a1f-45d1-8c3f-1066632f4f16 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:21.882049 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1007/2037] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:37:21 2022] PUT /identity/v3/projects/b06e471023194d10920aade4346c3bc3/users/6036528f63e3402283c4f3f96adf5f6e/roles/c51fe7c5ad804040879acfb54269455f => generated 0 bytes in 84 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:37:21.886043 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81143aa6-bc48-4e79-a255-6861b4925e43 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:21.896899 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81143aa6-bc48-4e79-a255-6861b4925e43 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:21.978078 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83237aef-39c4-4bd9-832a-d86929406736 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:21.979016 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83237aef-39c4-4bd9-832a-d86929406736 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:21.979278 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83237aef-39c4-4bd9-832a-d86929406736 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:21.979548 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83237aef-39c4-4bd9-832a-d86929406736 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:21.984806 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83237aef-39c4-4bd9-832a-d86929406736 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:21.986571 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83237aef-39c4-4bd9-832a-d86929406736 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:21.998994 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1031/2038] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:37:21 2022] GET /identity/v3/roles => generated 1420 bytes in 113 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 25 04:37:22.003452 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf74578f-1a1f-45d1-8c3f-1066632f4f16 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:22.016572 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf74578f-1a1f-45d1-8c3f-1066632f4f16 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:22.023562 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1c42962-6d2f-4c07-b9fb-3224828864d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:22.024210 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c42962-6d2f-4c07-b9fb-3224828864d9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:22.024396 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c42962-6d2f-4c07-b9fb-3224828864d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:22.024840 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c42962-6d2f-4c07-b9fb-3224828864d9 admin admin] PATH_INFO: `/v3/projects/b06e471023194d10920aade4346c3bc3/users/6036528f63e3402283c4f3f96adf5f6e/roles/c51fe7c5ad804040879acfb54269455f` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:22.027157 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1c42962-6d2f-4c07-b9fb-3224828864d9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b06e471023194d10920aade4346c3bc3, user_id=6036528f63e3402283c4f3f96adf5f6e, role_id=c51fe7c5ad804040879acfb54269455f)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:22.036967 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1c42962-6d2f-4c07-b9fb-3224828864d9 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:22.054803 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1008/2039] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:37:22 2022] PUT /identity/v3/projects/b06e471023194d10920aade4346c3bc3/users/6036528f63e3402283c4f3f96adf5f6e/roles/c51fe7c5ad804040879acfb54269455f => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:37:27.828807 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e150da3-5570-4a45-8d6b-188ccedab57c None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:27.829299 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e150da3-5570-4a45-8d6b-188ccedab57c None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:27.829376 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e150da3-5570-4a45-8d6b-188ccedab57c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:27.867048 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-7e150da3-5570-4a45-8d6b-188ccedab57c None None] MFA Rules not processed for user `6036528f63e3402283c4f3f96adf5f6e`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:37:27.868298 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-7e150da3-5570-4a45-8d6b-188ccedab57c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:37:27.878576 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:37:27.953147 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1c42962-6d2f-4c07-b9fb-3224828864d9 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:27.955440 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1032/2040] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:37:27 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:37:27.973055 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e150da3-5570-4a45-8d6b-188ccedab57c None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:27.974601 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-7e150da3-5570-4a45-8d6b-188ccedab57c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:37:28.078614 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1c42962-6d2f-4c07-b9fb-3224828864d9 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:28.083011 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e150da3-5570-4a45-8d6b-188ccedab57c None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:28.089312 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fa4f13b-a0c3-4c4f-9988-f6785ef125e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:28.090625 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fa4f13b-a0c3-4c4f-9988-f6785ef125e9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:28.090905 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fa4f13b-a0c3-4c4f-9988-f6785ef125e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:28.091177 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fa4f13b-a0c3-4c4f-9988-f6785ef125e9 admin admin] PATH_INFO: `/v3/users/51c1b359a50c4d87a79adc009204730d` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:28.092336 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fa4f13b-a0c3-4c4f-9988-f6785ef125e9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=51c1b359a50c4d87a79adc009204730d)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:28.092918 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b46d6925-fced-49c3-b976-2f9e21d70cca service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e703b42c53974daaa608b0ccc3f018b1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'ResellerAdmin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:28.093630 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b46d6925-fced-49c3-b976-2f9e21d70cca service nova] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:28.093975 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b46d6925-fced-49c3-b976-2f9e21d70cca service nova] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:28.094317 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b46d6925-fced-49c3-b976-2f9e21d70cca service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:28.094411 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fa4f13b-a0c3-4c4f-9988-f6785ef125e9 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:28.170011 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b46d6925-fced-49c3-b976-2f9e21d70cca service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:28.171588 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b46d6925-fced-49c3-b976-2f9e21d70cca service nova] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:28.185188 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1033/2041] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:37:27 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 214 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:37:28.185625 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-2fa4f13b-a0c3-4c4f-9988-f6785ef125e9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '51c1b359a50c4d87a79adc009204730d'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:28.194632 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-2fa4f13b-a0c3-4c4f-9988-f6785ef125e9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '51c1b359a50c4d87a79adc009204730d'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:28.211847 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-2fa4f13b-a0c3-4c4f-9988-f6785ef125e9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '51c1b359a50c4d87a79adc009204730d'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:28.213010 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-2fa4f13b-a0c3-4c4f-9988-f6785ef125e9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '51c1b359a50c4d87a79adc009204730d'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:28.223155 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-2fa4f13b-a0c3-4c4f-9988-f6785ef125e9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '51c1b359a50c4d87a79adc009204730d'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:28.230950 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1009/2042] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:37:27 2022] DELETE /identity/v3/users/51c1b359a50c4d87a79adc009204730d => generated 0 bytes in 277 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 04:37:28.235999 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-390d7fa2-abba-4894-89ba-76752d85cd98 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:28.236344 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-390d7fa2-abba-4894-89ba-76752d85cd98 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:28.236674 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-390d7fa2-abba-4894-89ba-76752d85cd98 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:28.260155 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-390d7fa2-abba-4894-89ba-76752d85cd98 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:37:28.261576 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-390d7fa2-abba-4894-89ba-76752d85cd98 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:37:28.380942 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:37:28.415629 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1034/2043] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:37:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:37:28.428250 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fa4f13b-a0c3-4c4f-9988-f6785ef125e9 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:28.431607 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-2fa4f13b-a0c3-4c4f-9988-f6785ef125e9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:37:28.512065 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:INSERT=4,keystone:SELECT=86,keystone:DELETE=10 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:37:28.542692 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fa4f13b-a0c3-4c4f-9988-f6785ef125e9 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:28.547547 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ea3ca9c-d803-4f8a-a453-af311554172d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:28.548223 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ea3ca9c-d803-4f8a-a453-af311554172d service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:28.548532 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ea3ca9c-d803-4f8a-a453-af311554172d service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:28.548928 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ea3ca9c-d803-4f8a-a453-af311554172d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:28.564490 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ea3ca9c-d803-4f8a-a453-af311554172d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:28.567735 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ea3ca9c-d803-4f8a-a453-af311554172d service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:28.587411 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1010/2044] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:37:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 160 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:37:29.204234 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-390d7fa2-abba-4894-89ba-76752d85cd98 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:29.205895 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-390d7fa2-abba-4894-89ba-76752d85cd98 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:37:29.218745 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-390d7fa2-abba-4894-89ba-76752d85cd98 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:29.224782 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e2078fc-5f0c-4b31-af6a-089c5aa5f6e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:29.225659 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e2078fc-5f0c-4b31-af6a-089c5aa5f6e3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:29.226030 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e2078fc-5f0c-4b31-af6a-089c5aa5f6e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:29.226370 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e2078fc-5f0c-4b31-af6a-089c5aa5f6e3 admin admin] PATH_INFO: `/v3/projects/901177d75ba64ca49037a6285e8c7b6a` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:29.227785 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e2078fc-5f0c-4b31-af6a-089c5aa5f6e3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=901177d75ba64ca49037a6285e8c7b6a)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:29.229916 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e2078fc-5f0c-4b31-af6a-089c5aa5f6e3 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:29.324629 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-2e2078fc-5f0c-4b31-af6a-089c5aa5f6e3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '901177d75ba64ca49037a6285e8c7b6a', 'request_id': 'req-2e2078fc-5f0c-4b31-af6a-089c5aa5f6e3'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:29.340478 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-2e2078fc-5f0c-4b31-af6a-089c5aa5f6e3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '901177d75ba64ca49037a6285e8c7b6a', 'request_id': 'req-2e2078fc-5f0c-4b31-af6a-089c5aa5f6e3'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:29.350734 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-2e2078fc-5f0c-4b31-af6a-089c5aa5f6e3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '901177d75ba64ca49037a6285e8c7b6a', 'request_id': 'req-2e2078fc-5f0c-4b31-af6a-089c5aa5f6e3'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:29.378222 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-2e2078fc-5f0c-4b31-af6a-089c5aa5f6e3 admin admin] The token cache is being invalidate because project 901177d75ba64ca49037a6285e8c7b6a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81434) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:37:29.378577 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-2e2078fc-5f0c-4b31-af6a-089c5aa5f6e3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:29.379446 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-2e2078fc-5f0c-4b31-af6a-089c5aa5f6e3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:29.381662 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1035/2045] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:37:29 2022] DELETE /identity/v3/projects/901177d75ba64ca49037a6285e8c7b6a => generated 0 bytes in 178 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:37:29.386987 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4d20179-fdf0-4e23-acae-6650a278cff8 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:29.387680 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4d20179-fdf0-4e23-acae-6650a278cff8 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:29.387854 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4d20179-fdf0-4e23-acae-6650a278cff8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:29.423632 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-b4d20179-fdf0-4e23-acae-6650a278cff8 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:37:29.425147 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-b4d20179-fdf0-4e23-acae-6650a278cff8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:37:29.545701 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1011/2046] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:37:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:37:29.602680 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f17f2554-c17a-41c8-9e34-46d1713b6c45 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:29.602971 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f17f2554-c17a-41c8-9e34-46d1713b6c45 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:29.603252 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f17f2554-c17a-41c8-9e34-46d1713b6c45 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:29.633187 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-f17f2554-c17a-41c8-9e34-46d1713b6c45 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:37:29.634609 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-f17f2554-c17a-41c8-9e34-46d1713b6c45 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:37:29.653541 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4d20179-fdf0-4e23-acae-6650a278cff8 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:29.660280 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1036/2047] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:37:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 57 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:37:29.664943 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4d20179-fdf0-4e23-acae-6650a278cff8 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:29.670661 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebb2b4c3-f94a-4835-92bb-5d7aa15b754a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:29.671447 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb2b4c3-f94a-4835-92bb-5d7aa15b754a admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:29.671834 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb2b4c3-f94a-4835-92bb-5d7aa15b754a admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:29.672231 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb2b4c3-f94a-4835-92bb-5d7aa15b754a admin admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:29.673619 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebb2b4c3-f94a-4835-92bb-5d7aa15b754a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:37:29.673619 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebb2b4c3-f94a-4835-92bb-5d7aa15b754a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:29.675400 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebb2b4c3-f94a-4835-92bb-5d7aa15b754a admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:29.698677 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f17f2554-c17a-41c8-9e34-46d1713b6c45 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:29.700904 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1012/2048] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Sep 25 04:37:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:37:29.721235 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f17f2554-c17a-41c8-9e34-46d1713b6c45 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:29.728425 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe02c431-0760-4f25-8987-20049c6fd719 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:29.729804 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe02c431-0760-4f25-8987-20049c6fd719 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:29.730157 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe02c431-0760-4f25-8987-20049c6fd719 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:29.732427 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe02c431-0760-4f25-8987-20049c6fd719 admin admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:29.733418 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe02c431-0760-4f25-8987-20049c6fd719 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:29.738135 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe02c431-0760-4f25-8987-20049c6fd719 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:29.804218 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1037/2049] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:37:29 2022] POST /identity/v3/projects => generated 375 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:37:29.805145 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebb2b4c3-f94a-4835-92bb-5d7aa15b754a admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:29.825779 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebb2b4c3-f94a-4835-92bb-5d7aa15b754a admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:29.954333 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b438adc-88e5-4b27-bac7-fa3576016ee1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:29.955158 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b438adc-88e5-4b27-bac7-fa3576016ee1 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:29.955584 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b438adc-88e5-4b27-bac7-fa3576016ee1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:29.956074 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b438adc-88e5-4b27-bac7-fa3576016ee1 admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:29.957097 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b438adc-88e5-4b27-bac7-fa3576016ee1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:29.959748 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b438adc-88e5-4b27-bac7-fa3576016ee1 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:30.014675 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe02c431-0760-4f25-8987-20049c6fd719 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:30.014893 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1013/2050] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:37:29 2022] POST /identity/v3/users => generated 288 bytes in 206 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:37:30.029872 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe02c431-0760-4f25-8987-20049c6fd719 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:30.043966 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1aff01d-0729-4090-a191-9641c993d5c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:30.044619 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1aff01d-0729-4090-a191-9641c993d5c5 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:30.044955 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1aff01d-0729-4090-a191-9641c993d5c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:30.045224 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1aff01d-0729-4090-a191-9641c993d5c5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:30.045870 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1aff01d-0729-4090-a191-9641c993d5c5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:30.047463 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1aff01d-0729-4090-a191-9641c993d5c5 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:30.062063 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1038/2051] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:37:30 2022] GET /identity/v3/roles => generated 1420 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:37:30.065959 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b438adc-88e5-4b27-bac7-fa3576016ee1 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:30.079637 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b438adc-88e5-4b27-bac7-fa3576016ee1 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:30.084833 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b47fa1f8-2311-4fb0-9daf-486e66854cca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:30.085548 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47fa1f8-2311-4fb0-9daf-486e66854cca admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:30.086079 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47fa1f8-2311-4fb0-9daf-486e66854cca admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:30.086454 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47fa1f8-2311-4fb0-9daf-486e66854cca admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:30.087194 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b47fa1f8-2311-4fb0-9daf-486e66854cca admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:30.088925 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b47fa1f8-2311-4fb0-9daf-486e66854cca admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:30.102415 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1014/2052] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:37:30 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:37:30.106762 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1aff01d-0729-4090-a191-9641c993d5c5 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:30.119023 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1aff01d-0729-4090-a191-9641c993d5c5 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:30.125288 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f69e9df1-23ab-40f4-a9a8-d668f5c3a5ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:30.127297 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f69e9df1-23ab-40f4-a9a8-d668f5c3a5ea admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:30.127297 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f69e9df1-23ab-40f4-a9a8-d668f5c3a5ea admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:30.127593 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f69e9df1-23ab-40f4-a9a8-d668f5c3a5ea admin admin] PATH_INFO: `/v3/projects/1c7b3c70b5ae40f092a9a009b3161b9b/users/a5dc5e5e9bb34eb7a3242f169828ddfa/roles/8f6cc574056c4278bc9778f14dc4b411` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:30.149862 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f69e9df1-23ab-40f4-a9a8-d668f5c3a5ea admin admin] RBAC: Authorizing `identity:create_grant(project_id=1c7b3c70b5ae40f092a9a009b3161b9b, user_id=a5dc5e5e9bb34eb7a3242f169828ddfa, role_id=8f6cc574056c4278bc9778f14dc4b411)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:30.153808 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f69e9df1-23ab-40f4-a9a8-d668f5c3a5ea admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:30.172129 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1039/2053] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:37:30 2022] PUT /identity/v3/projects/1c7b3c70b5ae40f092a9a009b3161b9b/users/a5dc5e5e9bb34eb7a3242f169828ddfa/roles/8f6cc574056c4278bc9778f14dc4b411 => generated 0 bytes in 66 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:37:30.182429 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea894715-717b-4681-8e93-d7db14fdb159 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:30.182712 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea894715-717b-4681-8e93-d7db14fdb159 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:30.182985 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea894715-717b-4681-8e93-d7db14fdb159 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:30.214588 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-ea894715-717b-4681-8e93-d7db14fdb159 None None] MFA Rules not processed for user `a5dc5e5e9bb34eb7a3242f169828ddfa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:37:30.216058 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-ea894715-717b-4681-8e93-d7db14fdb159 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:37:30.288692 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:37:30.369355 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1015/2054] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:37:30 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:37:30.369978 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f69e9df1-23ab-40f4-a9a8-d668f5c3a5ea admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:30.491053 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f69e9df1-23ab-40f4-a9a8-d668f5c3a5ea admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:30.497598 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a21084a-be30-4a81-bf9b-f6e7336eb642 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:30.498368 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a21084a-be30-4a81-bf9b-f6e7336eb642 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:30.498648 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a21084a-be30-4a81-bf9b-f6e7336eb642 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:30.498941 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a21084a-be30-4a81-bf9b-f6e7336eb642 admin admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:30.499541 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a21084a-be30-4a81-bf9b-f6e7336eb642 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:30.502509 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a21084a-be30-4a81-bf9b-f6e7336eb642 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:30.529038 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:37:30.563931 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea894715-717b-4681-8e93-d7db14fdb159 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:30.565625 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1040/2055] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:37:30 2022] POST /identity/v3/projects => generated 377 bytes in 190 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:37:30.579251 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea894715-717b-4681-8e93-d7db14fdb159 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:30.734569 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8881ab82-d9d4-4e2a-b112-23a214f6c952 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:30.735035 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8881ab82-d9d4-4e2a-b112-23a214f6c952 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:30.735286 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8881ab82-d9d4-4e2a-b112-23a214f6c952 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:30.735557 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8881ab82-d9d4-4e2a-b112-23a214f6c952 admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:30.736983 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8881ab82-d9d4-4e2a-b112-23a214f6c952 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:30.738153 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8881ab82-d9d4-4e2a-b112-23a214f6c952 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:30.775771 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1016/2056] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:37:30 2022] POST /identity/v3/users => generated 289 bytes in 210 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:37:30.777315 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a21084a-be30-4a81-bf9b-f6e7336eb642 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:30.794074 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a21084a-be30-4a81-bf9b-f6e7336eb642 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:30.809800 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be7eb7e9-406e-4ccf-8a6e-907143d48ed3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:30.810443 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be7eb7e9-406e-4ccf-8a6e-907143d48ed3 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:30.811083 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be7eb7e9-406e-4ccf-8a6e-907143d48ed3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:30.811210 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be7eb7e9-406e-4ccf-8a6e-907143d48ed3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:30.811797 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be7eb7e9-406e-4ccf-8a6e-907143d48ed3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:30.813486 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be7eb7e9-406e-4ccf-8a6e-907143d48ed3 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:30.824689 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1041/2057] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:37:30 2022] GET /identity/v3/roles => generated 1420 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:37:30.829609 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8881ab82-d9d4-4e2a-b112-23a214f6c952 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:30.844042 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8881ab82-d9d4-4e2a-b112-23a214f6c952 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:30.853469 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70fee169-cf11-46a2-b5b4-2dbbb7385137 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:30.854482 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70fee169-cf11-46a2-b5b4-2dbbb7385137 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:30.854963 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70fee169-cf11-46a2-b5b4-2dbbb7385137 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:30.857343 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70fee169-cf11-46a2-b5b4-2dbbb7385137 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:30.857865 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70fee169-cf11-46a2-b5b4-2dbbb7385137 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:30.859455 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70fee169-cf11-46a2-b5b4-2dbbb7385137 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:30.884380 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1017/2058] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:37:30 2022] GET /identity/v3/roles => generated 1420 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 25 04:37:30.888729 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be7eb7e9-406e-4ccf-8a6e-907143d48ed3 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:30.903421 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be7eb7e9-406e-4ccf-8a6e-907143d48ed3 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:30.912975 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01d28020-3035-4c1d-99a0-88a46a76e9fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:30.913662 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01d28020-3035-4c1d-99a0-88a46a76e9fa admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:30.913947 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01d28020-3035-4c1d-99a0-88a46a76e9fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:30.914231 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01d28020-3035-4c1d-99a0-88a46a76e9fa admin admin] PATH_INFO: `/v3/projects/01106f395c3f4512959f7f967dcfd82b/users/d44e6619cda649bf9cdc586139249854/roles/8f6cc574056c4278bc9778f14dc4b411` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:30.947828 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01d28020-3035-4c1d-99a0-88a46a76e9fa admin admin] RBAC: Authorizing `identity:create_grant(project_id=01106f395c3f4512959f7f967dcfd82b, user_id=d44e6619cda649bf9cdc586139249854, role_id=8f6cc574056c4278bc9778f14dc4b411)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:30.952012 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01d28020-3035-4c1d-99a0-88a46a76e9fa admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:30.973134 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1042/2059] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:37:30 2022] PUT /identity/v3/projects/01106f395c3f4512959f7f967dcfd82b/users/d44e6619cda649bf9cdc586139249854/roles/8f6cc574056c4278bc9778f14dc4b411 => generated 0 bytes in 85 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:37:30.985376 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69fae90f-6479-49bc-92d3-faef2ed3632c None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:30.985672 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69fae90f-6479-49bc-92d3-faef2ed3632c None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:30.985958 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69fae90f-6479-49bc-92d3-faef2ed3632c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:31.069062 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-69fae90f-6479-49bc-92d3-faef2ed3632c None None] MFA Rules not processed for user `d44e6619cda649bf9cdc586139249854`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:37:31.070682 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-69fae90f-6479-49bc-92d3-faef2ed3632c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:37:31.218454 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01d28020-3035-4c1d-99a0-88a46a76e9fa admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:31.219167 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1018/2060] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:37:30 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 235 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:37:31.323563 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01d28020-3035-4c1d-99a0-88a46a76e9fa admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:31.329902 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19f38bdc-142d-4e39-888e-557906f62117 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:31.330542 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19f38bdc-142d-4e39-888e-557906f62117 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:31.331109 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19f38bdc-142d-4e39-888e-557906f62117 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:31.332047 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19f38bdc-142d-4e39-888e-557906f62117 admin admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:31.333087 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19f38bdc-142d-4e39-888e-557906f62117 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:31.335804 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19f38bdc-142d-4e39-888e-557906f62117 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:31.378205 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1043/2061] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:37:31 2022] POST /identity/v3/projects => generated 377 bytes in 158 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:37:31.381357 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69fae90f-6479-49bc-92d3-faef2ed3632c None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:31.396445 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69fae90f-6479-49bc-92d3-faef2ed3632c None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:31.448949 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:37:31.475872 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f920674e-d790-4e1d-ada9-dec14afe9958 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:31.476948 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f920674e-d790-4e1d-ada9-dec14afe9958 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:31.477575 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f920674e-d790-4e1d-ada9-dec14afe9958 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:31.478161 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f920674e-d790-4e1d-ada9-dec14afe9958 admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:31.479027 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f920674e-d790-4e1d-ada9-dec14afe9958 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:31.482317 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f920674e-d790-4e1d-ada9-dec14afe9958 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:31.509186 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1019/2062] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:37:31 2022] POST /identity/v3/users => generated 295 bytes in 129 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:37:31.513114 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19f38bdc-142d-4e39-888e-557906f62117 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:31.526374 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19f38bdc-142d-4e39-888e-557906f62117 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:31.551549 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54d5effe-ab97-4b59-a3cb-35eb7772f5ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:31.552260 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d5effe-ab97-4b59-a3cb-35eb7772f5ab admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:31.552706 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d5effe-ab97-4b59-a3cb-35eb7772f5ab admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:31.553289 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d5effe-ab97-4b59-a3cb-35eb7772f5ab admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:31.554116 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54d5effe-ab97-4b59-a3cb-35eb7772f5ab admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:31.556755 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54d5effe-ab97-4b59-a3cb-35eb7772f5ab admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:31.583769 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1044/2063] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:37:31 2022] GET /identity/v3/roles => generated 1420 bytes in 71 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 25 04:37:31.588084 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f920674e-d790-4e1d-ada9-dec14afe9958 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:31.604290 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f920674e-d790-4e1d-ada9-dec14afe9958 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:31.619257 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dba8bb28-2ea8-4069-9aef-b50f1232a38c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:31.622688 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dba8bb28-2ea8-4069-9aef-b50f1232a38c admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:31.623045 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dba8bb28-2ea8-4069-9aef-b50f1232a38c admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:31.624234 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dba8bb28-2ea8-4069-9aef-b50f1232a38c admin admin] PATH_INFO: `/v3/projects/fa429cb2cdd2454291b51b175d1b15fe/users/8810f783fc55406f9bf2ca0b0bcd6d1b/roles/c51fe7c5ad804040879acfb54269455f` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:31.653114 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dba8bb28-2ea8-4069-9aef-b50f1232a38c admin admin] RBAC: Authorizing `identity:create_grant(project_id=fa429cb2cdd2454291b51b175d1b15fe, user_id=8810f783fc55406f9bf2ca0b0bcd6d1b, role_id=c51fe7c5ad804040879acfb54269455f)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:31.657127 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dba8bb28-2ea8-4069-9aef-b50f1232a38c admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:31.679776 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1020/2064] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:37:31 2022] PUT /identity/v3/projects/fa429cb2cdd2454291b51b175d1b15fe/users/8810f783fc55406f9bf2ca0b0bcd6d1b/roles/c51fe7c5ad804040879acfb54269455f => generated 0 bytes in 92 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:37:31.684755 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54d5effe-ab97-4b59-a3cb-35eb7772f5ab admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:31.702548 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54d5effe-ab97-4b59-a3cb-35eb7772f5ab admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:31.827568 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-050e82f8-922f-4518-9074-f995a6b3e334 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:31.827568 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-050e82f8-922f-4518-9074-f995a6b3e334 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:31.827568 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-050e82f8-922f-4518-9074-f995a6b3e334 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:31.834547 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-050e82f8-922f-4518-9074-f995a6b3e334 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:31.838550 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-050e82f8-922f-4518-9074-f995a6b3e334 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:31.842878 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-050e82f8-922f-4518-9074-f995a6b3e334 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:31.872126 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1045/2065] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:37:31 2022] GET /identity/v3/roles => generated 1420 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:37:31.874867 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dba8bb28-2ea8-4069-9aef-b50f1232a38c admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:31.892940 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dba8bb28-2ea8-4069-9aef-b50f1232a38c admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:31.899411 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86a33b4e-c980-41e8-be17-3dfb99529adb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:31.900166 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a33b4e-c980-41e8-be17-3dfb99529adb admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:31.900452 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a33b4e-c980-41e8-be17-3dfb99529adb admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:31.900763 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a33b4e-c980-41e8-be17-3dfb99529adb admin admin] PATH_INFO: `/v3/projects/fa429cb2cdd2454291b51b175d1b15fe/users/8810f783fc55406f9bf2ca0b0bcd6d1b/roles/c51fe7c5ad804040879acfb54269455f` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:31.903502 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86a33b4e-c980-41e8-be17-3dfb99529adb admin admin] RBAC: Authorizing `identity:create_grant(project_id=fa429cb2cdd2454291b51b175d1b15fe, user_id=8810f783fc55406f9bf2ca0b0bcd6d1b, role_id=c51fe7c5ad804040879acfb54269455f)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:31.906433 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86a33b4e-c980-41e8-be17-3dfb99529adb admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:31.928770 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1021/2066] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:37:31 2022] PUT /identity/v3/projects/fa429cb2cdd2454291b51b175d1b15fe/users/8810f783fc55406f9bf2ca0b0bcd6d1b/roles/c51fe7c5ad804040879acfb54269455f => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:37:31.942651 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c959339-7c22-41c8-8451-fa96823a9bc7 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:31.943122 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c959339-7c22-41c8-8451-fa96823a9bc7 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:31.943580 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c959339-7c22-41c8-8451-fa96823a9bc7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:31.987162 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-7c959339-7c22-41c8-8451-fa96823a9bc7 None None] MFA Rules not processed for user `8810f783fc55406f9bf2ca0b0bcd6d1b`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:37:31.988391 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-7c959339-7c22-41c8-8451-fa96823a9bc7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:37:32.015522 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:37:32.168690 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1046/2067] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:37:31 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 226 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:37:32.179863 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86a33b4e-c980-41e8-be17-3dfb99529adb admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:32.183380 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-86a33b4e-c980-41e8-be17-3dfb99529adb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:37:32.297953 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86a33b4e-c980-41e8-be17-3dfb99529adb admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:32.302687 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b386bd74-3d89-4046-b510-16bc83ce81dd service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ffbc136c77b84e8c87128f7247f4ec03', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:32.303351 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b386bd74-3d89-4046-b510-16bc83ce81dd service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:32.303631 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b386bd74-3d89-4046-b510-16bc83ce81dd service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:32.303910 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b386bd74-3d89-4046-b510-16bc83ce81dd service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:32.380462 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c959339-7c22-41c8-8451-fa96823a9bc7 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:32.382331 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-7c959339-7c22-41c8-8451-fa96823a9bc7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:37:32.409386 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b386bd74-3d89-4046-b510-16bc83ce81dd service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:32.411139 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b386bd74-3d89-4046-b510-16bc83ce81dd service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:32.435709 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1022/2068] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:37:32 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 256 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:37:32.486988 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c959339-7c22-41c8-8451-fa96823a9bc7 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:32.494313 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c02ff95a-3b93-4333-ae59-9eb3e41fa04e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:32.495161 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c02ff95a-3b93-4333-ae59-9eb3e41fa04e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:32.495682 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c02ff95a-3b93-4333-ae59-9eb3e41fa04e admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:32.496164 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c02ff95a-3b93-4333-ae59-9eb3e41fa04e admin admin] PATH_INFO: `/v3/users/848b4af60f984dc788ae6994e89e4acf` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:32.497665 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c02ff95a-3b93-4333-ae59-9eb3e41fa04e admin admin] RBAC: Authorizing `identity:delete_user(user_id=848b4af60f984dc788ae6994e89e4acf)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:32.500098 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c02ff95a-3b93-4333-ae59-9eb3e41fa04e admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:32.578483 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-c02ff95a-3b93-4333-ae59-9eb3e41fa04e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '848b4af60f984dc788ae6994e89e4acf'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:32.593672 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-c02ff95a-3b93-4333-ae59-9eb3e41fa04e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '848b4af60f984dc788ae6994e89e4acf'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:32.625184 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-c02ff95a-3b93-4333-ae59-9eb3e41fa04e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '848b4af60f984dc788ae6994e89e4acf'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:32.625946 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-c02ff95a-3b93-4333-ae59-9eb3e41fa04e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '848b4af60f984dc788ae6994e89e4acf'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:32.640714 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-c02ff95a-3b93-4333-ae59-9eb3e41fa04e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '848b4af60f984dc788ae6994e89e4acf'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:32.660263 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1047/2069] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:37:32 2022] DELETE /identity/v3/users/848b4af60f984dc788ae6994e89e4acf => generated 0 bytes in 281 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:37:32.665435 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc968f31-cf59-464c-b797-1b15e848fd37 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:32.666225 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc968f31-cf59-464c-b797-1b15e848fd37 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:32.666561 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc968f31-cf59-464c-b797-1b15e848fd37 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:32.695853 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-bc968f31-cf59-464c-b797-1b15e848fd37 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:37:32.697306 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-bc968f31-cf59-464c-b797-1b15e848fd37 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:37:32.761038 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:37:32.783785 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1023/2070] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:37:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:37:32.813813 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c02ff95a-3b93-4333-ae59-9eb3e41fa04e admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:32.814584 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-c02ff95a-3b93-4333-ae59-9eb3e41fa04e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:37:32.873089 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:37:32.972592 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c02ff95a-3b93-4333-ae59-9eb3e41fa04e admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:32.986038 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42b77a90-a92c-40a8-b5ca-1413037d5bed service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:32.986798 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42b77a90-a92c-40a8-b5ca-1413037d5bed service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:32.987799 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42b77a90-a92c-40a8-b5ca-1413037d5bed service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:32.988028 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42b77a90-a92c-40a8-b5ca-1413037d5bed service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:33.018271 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42b77a90-a92c-40a8-b5ca-1413037d5bed service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:33.029828 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42b77a90-a92c-40a8-b5ca-1413037d5bed service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:33.048272 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1048/2071] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:37:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 237 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 25 04:37:33.612552 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc968f31-cf59-464c-b797-1b15e848fd37 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:33.614215 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-bc968f31-cf59-464c-b797-1b15e848fd37 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:37:33.645877 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc968f31-cf59-464c-b797-1b15e848fd37 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:33.657820 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99997877-9f60-4f34-a65b-be160e449285 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:33.658493 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99997877-9f60-4f34-a65b-be160e449285 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:33.658944 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99997877-9f60-4f34-a65b-be160e449285 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:33.659137 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99997877-9f60-4f34-a65b-be160e449285 admin admin] PATH_INFO: `/v3/projects/b6a625eeaecd4610906f007a4174fc0a` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:33.660423 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99997877-9f60-4f34-a65b-be160e449285 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b6a625eeaecd4610906f007a4174fc0a)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:33.662683 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99997877-9f60-4f34-a65b-be160e449285 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:33.726823 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-99997877-9f60-4f34-a65b-be160e449285 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b6a625eeaecd4610906f007a4174fc0a', 'request_id': 'req-99997877-9f60-4f34-a65b-be160e449285'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:33.735025 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-99997877-9f60-4f34-a65b-be160e449285 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b6a625eeaecd4610906f007a4174fc0a', 'request_id': 'req-99997877-9f60-4f34-a65b-be160e449285'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:33.742209 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-99997877-9f60-4f34-a65b-be160e449285 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b6a625eeaecd4610906f007a4174fc0a', 'request_id': 'req-99997877-9f60-4f34-a65b-be160e449285'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:33.763695 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-99997877-9f60-4f34-a65b-be160e449285 admin admin] The token cache is being invalidate because project b6a625eeaecd4610906f007a4174fc0a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:37:33.763991 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-99997877-9f60-4f34-a65b-be160e449285 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:33.764803 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-99997877-9f60-4f34-a65b-be160e449285 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:33.766646 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1024/2072] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:37:33 2022] DELETE /identity/v3/projects/b6a625eeaecd4610906f007a4174fc0a => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:37:33.770402 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42b77a90-a92c-40a8-b5ca-1413037d5bed service neutron] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:33.781327 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42b77a90-a92c-40a8-b5ca-1413037d5bed service neutron] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:33.858729 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfecf7fb-7ce8-4ab3-ba37-12ada7f64108 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:33.859382 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfecf7fb-7ce8-4ab3-ba37-12ada7f64108 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:33.859659 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfecf7fb-7ce8-4ab3-ba37-12ada7f64108 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:33.859927 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfecf7fb-7ce8-4ab3-ba37-12ada7f64108 admin admin] PATH_INFO: `/v3/users/30d27a152fd845e1806a39293f3a8ffe` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:33.861238 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfecf7fb-7ce8-4ab3-ba37-12ada7f64108 admin admin] RBAC: Authorizing `identity:delete_user(user_id=30d27a152fd845e1806a39293f3a8ffe)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:33.864455 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfecf7fb-7ce8-4ab3-ba37-12ada7f64108 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:33.932494 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-cfecf7fb-7ce8-4ab3-ba37-12ada7f64108 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '30d27a152fd845e1806a39293f3a8ffe'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:33.942819 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-cfecf7fb-7ce8-4ab3-ba37-12ada7f64108 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '30d27a152fd845e1806a39293f3a8ffe'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:33.967771 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-cfecf7fb-7ce8-4ab3-ba37-12ada7f64108 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '30d27a152fd845e1806a39293f3a8ffe'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:33.968607 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-cfecf7fb-7ce8-4ab3-ba37-12ada7f64108 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '30d27a152fd845e1806a39293f3a8ffe'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:33.975970 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-cfecf7fb-7ce8-4ab3-ba37-12ada7f64108 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '30d27a152fd845e1806a39293f3a8ffe'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:33.982610 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1049/2073] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:37:33 2022] DELETE /identity/v3/users/30d27a152fd845e1806a39293f3a8ffe => generated 0 bytes in 213 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 04:37:34.460104 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99997877-9f60-4f34-a65b-be160e449285 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:34.461571 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-99997877-9f60-4f34-a65b-be160e449285 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:37:34.483351 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99997877-9f60-4f34-a65b-be160e449285 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:34.575406 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bf50c9b-5046-4054-9196-a58aebc77e39 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:34.576071 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf50c9b-5046-4054-9196-a58aebc77e39 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:34.576353 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf50c9b-5046-4054-9196-a58aebc77e39 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:34.576669 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf50c9b-5046-4054-9196-a58aebc77e39 admin admin] PATH_INFO: `/v3/projects/b0ae2308c6424b6db559e7389154b104` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:34.577831 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bf50c9b-5046-4054-9196-a58aebc77e39 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b0ae2308c6424b6db559e7389154b104)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:34.579977 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bf50c9b-5046-4054-9196-a58aebc77e39 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:34.643902 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-1bf50c9b-5046-4054-9196-a58aebc77e39 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b0ae2308c6424b6db559e7389154b104', 'request_id': 'req-1bf50c9b-5046-4054-9196-a58aebc77e39'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:34.652221 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-1bf50c9b-5046-4054-9196-a58aebc77e39 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b0ae2308c6424b6db559e7389154b104', 'request_id': 'req-1bf50c9b-5046-4054-9196-a58aebc77e39'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:34.659512 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-1bf50c9b-5046-4054-9196-a58aebc77e39 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b0ae2308c6424b6db559e7389154b104', 'request_id': 'req-1bf50c9b-5046-4054-9196-a58aebc77e39'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:34.674594 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-1bf50c9b-5046-4054-9196-a58aebc77e39 admin admin] The token cache is being invalidate because project b0ae2308c6424b6db559e7389154b104 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:37:34.675339 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-1bf50c9b-5046-4054-9196-a58aebc77e39 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:34.675579 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-1bf50c9b-5046-4054-9196-a58aebc77e39 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:34.677454 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1025/2074] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:37:34 2022] DELETE /identity/v3/projects/b0ae2308c6424b6db559e7389154b104 => generated 0 bytes in 218 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:37:34.682720 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c15504-197c-444c-ace9-43d9bc7ad155 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:34.683207 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c15504-197c-444c-ace9-43d9bc7ad155 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:34.683574 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c15504-197c-444c-ace9-43d9bc7ad155 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:34.711039 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-02c15504-197c-444c-ace9-43d9bc7ad155 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:37:34.722852 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-02c15504-197c-444c-ace9-43d9bc7ad155 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:37:34.790624 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:37:34.811498 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1050/2075] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:37:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 04:37:34.820450 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e95dc3f-1bd4-44bf-a27a-82264903df2c None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:34.821434 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e95dc3f-1bd4-44bf-a27a-82264903df2c None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:34.821719 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e95dc3f-1bd4-44bf-a27a-82264903df2c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:34.834272 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:37:34.853064 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-8e95dc3f-1bd4-44bf-a27a-82264903df2c None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:37:34.854595 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-8e95dc3f-1bd4-44bf-a27a-82264903df2c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:37:34.867665 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1026/2076] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:37:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 48 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:37:34.868826 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02c15504-197c-444c-ace9-43d9bc7ad155 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:34.884322 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02c15504-197c-444c-ace9-43d9bc7ad155 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:34.891189 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35cea473-ed26-4c8f-8766-ee5297ba7a76 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:34.891805 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35cea473-ed26-4c8f-8766-ee5297ba7a76 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:34.892163 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35cea473-ed26-4c8f-8766-ee5297ba7a76 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:34.892342 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35cea473-ed26-4c8f-8766-ee5297ba7a76 admin admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:34.893055 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35cea473-ed26-4c8f-8766-ee5297ba7a76 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:37:34.893356 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35cea473-ed26-4c8f-8766-ee5297ba7a76 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:34.894955 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35cea473-ed26-4c8f-8766-ee5297ba7a76 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:34.915102 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1051/2077] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Sep 25 04:37:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:37:34.919572 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e95dc3f-1bd4-44bf-a27a-82264903df2c None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:34.931113 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e95dc3f-1bd4-44bf-a27a-82264903df2c None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:34.937836 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be503f00-7743-4c29-ada7-4c0f4dcd15b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:34.938776 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be503f00-7743-4c29-ada7-4c0f4dcd15b4 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:34.939211 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be503f00-7743-4c29-ada7-4c0f4dcd15b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:34.939545 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be503f00-7743-4c29-ada7-4c0f4dcd15b4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:34.940260 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be503f00-7743-4c29-ada7-4c0f4dcd15b4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:34.942523 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be503f00-7743-4c29-ada7-4c0f4dcd15b4 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:34.980675 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1027/2078] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:37:34 2022] POST /identity/v3/projects => generated 407 bytes in 59 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:37:34.982146 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35cea473-ed26-4c8f-8766-ee5297ba7a76 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:34.994867 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35cea473-ed26-4c8f-8766-ee5297ba7a76 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:35.073732 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38b08ace-a735-46ed-8917-94cfcbf0498d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:35.074475 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38b08ace-a735-46ed-8917-94cfcbf0498d admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:35.074752 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38b08ace-a735-46ed-8917-94cfcbf0498d admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:35.075238 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38b08ace-a735-46ed-8917-94cfcbf0498d admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:35.076186 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38b08ace-a735-46ed-8917-94cfcbf0498d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:35.078152 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38b08ace-a735-46ed-8917-94cfcbf0498d admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:35.104241 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1052/2079] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:37:34 2022] POST /identity/v3/users => generated 304 bytes in 122 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:37:35.108996 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be503f00-7743-4c29-ada7-4c0f4dcd15b4 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:35.120478 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be503f00-7743-4c29-ada7-4c0f4dcd15b4 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:35.129821 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-442c0368-313c-4c97-83e6-671b10875af9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:35.130463 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-442c0368-313c-4c97-83e6-671b10875af9 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:35.131114 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-442c0368-313c-4c97-83e6-671b10875af9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:35.131404 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-442c0368-313c-4c97-83e6-671b10875af9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:35.132453 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-442c0368-313c-4c97-83e6-671b10875af9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:35.134135 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-442c0368-313c-4c97-83e6-671b10875af9 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:35.156078 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1028/2080] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:37:35 2022] GET /identity/v3/roles => generated 1420 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:37:35.159145 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38b08ace-a735-46ed-8917-94cfcbf0498d admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:35.171462 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38b08ace-a735-46ed-8917-94cfcbf0498d admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:35.177591 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09c4b216-02de-4557-8cbb-d597053d6188 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:35.178267 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c4b216-02de-4557-8cbb-d597053d6188 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:35.178557 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c4b216-02de-4557-8cbb-d597053d6188 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:35.178829 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c4b216-02de-4557-8cbb-d597053d6188 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:35.179483 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09c4b216-02de-4557-8cbb-d597053d6188 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:35.181294 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09c4b216-02de-4557-8cbb-d597053d6188 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:35.196219 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1053/2081] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:37:35 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:37:35.197356 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-442c0368-313c-4c97-83e6-671b10875af9 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:35.212111 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-442c0368-313c-4c97-83e6-671b10875af9 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:35.217829 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aa2b2db-f76c-44c5-b873-bea6c4bd642e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:35.218930 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aa2b2db-f76c-44c5-b873-bea6c4bd642e admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:35.219215 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aa2b2db-f76c-44c5-b873-bea6c4bd642e admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:35.219328 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09c4b216-02de-4557-8cbb-d597053d6188 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:35.219603 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aa2b2db-f76c-44c5-b873-bea6c4bd642e admin admin] PATH_INFO: `/v3/projects/8c82c76db2be48ea95bd8e55de046a95/users/71461b6fab73487b84d40cd31e3c8b9d/roles/8f6cc574056c4278bc9778f14dc4b411` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:35.221024 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-09c4b216-02de-4557-8cbb-d597053d6188 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:37:35.236445 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7aa2b2db-f76c-44c5-b873-bea6c4bd642e admin admin] RBAC: Authorizing `identity:create_grant(project_id=8c82c76db2be48ea95bd8e55de046a95, user_id=71461b6fab73487b84d40cd31e3c8b9d, role_id=8f6cc574056c4278bc9778f14dc4b411)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:35.239205 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7aa2b2db-f76c-44c5-b873-bea6c4bd642e admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:35.255031 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1029/2082] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:37:35 2022] PUT /identity/v3/projects/8c82c76db2be48ea95bd8e55de046a95/users/71461b6fab73487b84d40cd31e3c8b9d/roles/8f6cc574056c4278bc9778f14dc4b411 => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:37:35.266150 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93b5123e-da94-461d-ac06-5c4706182289 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:35.266439 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93b5123e-da94-461d-ac06-5c4706182289 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:35.266702 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93b5123e-da94-461d-ac06-5c4706182289 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:35.285579 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09c4b216-02de-4557-8cbb-d597053d6188 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:35.298476 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-93b5123e-da94-461d-ac06-5c4706182289 None None] MFA Rules not processed for user `71461b6fab73487b84d40cd31e3c8b9d`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:37:35.299631 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-93b5123e-da94-461d-ac06-5c4706182289 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:37:35.344155 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dbac9bb-f18e-4682-a611-5f887299fc6d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ffbc136c77b84e8c87128f7247f4ec03', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:35.345037 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dbac9bb-f18e-4682-a611-5f887299fc6d service cinder] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:35.345317 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dbac9bb-f18e-4682-a611-5f887299fc6d service cinder] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:35.345599 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dbac9bb-f18e-4682-a611-5f887299fc6d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:35.347381 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-9dbac9bb-f18e-4682-a611-5f887299fc6d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:37:35.374844 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:37:35.392939 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1030/2083] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:37:35 2022] POST /identity/v3/auth/tokens => generated 3284 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 04:37:35.396179 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93b5123e-da94-461d-ac06-5c4706182289 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:35.413016 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dbac9bb-f18e-4682-a611-5f887299fc6d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:35.414738 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dbac9bb-f18e-4682-a611-5f887299fc6d service cinder] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:35.424811 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1054/2084] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:37:35 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 206 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:37:35.466637 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93b5123e-da94-461d-ac06-5c4706182289 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:35.471887 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e6e6c3a-fc29-4df3-a7f8-69ea3acd8b17 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:35.472542 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e6e6c3a-fc29-4df3-a7f8-69ea3acd8b17 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:35.472872 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e6e6c3a-fc29-4df3-a7f8-69ea3acd8b17 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:35.473181 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e6e6c3a-fc29-4df3-a7f8-69ea3acd8b17 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:35.473825 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e6e6c3a-fc29-4df3-a7f8-69ea3acd8b17 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:35.475769 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e6e6c3a-fc29-4df3-a7f8-69ea3acd8b17 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:35.507728 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1031/2085] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:37:35 2022] POST /identity/v3/projects => generated 409 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 04:37:35.512659 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dbac9bb-f18e-4682-a611-5f887299fc6d service cinder] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:35.524184 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dbac9bb-f18e-4682-a611-5f887299fc6d service cinder] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:35.601131 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd10b68a-b7f3-41d6-aa1d-c570c2b2f96b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:35.601753 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd10b68a-b7f3-41d6-aa1d-c570c2b2f96b admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:35.602033 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd10b68a-b7f3-41d6-aa1d-c570c2b2f96b admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:35.602307 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd10b68a-b7f3-41d6-aa1d-c570c2b2f96b admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:35.603010 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd10b68a-b7f3-41d6-aa1d-c570c2b2f96b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:35.605462 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd10b68a-b7f3-41d6-aa1d-c570c2b2f96b admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:35.629179 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1055/2086] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:37:35 2022] POST /identity/v3/users => generated 311 bytes in 118 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:37:35.633244 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e6e6c3a-fc29-4df3-a7f8-69ea3acd8b17 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:35.644261 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e6e6c3a-fc29-4df3-a7f8-69ea3acd8b17 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:35.649644 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-417b0c77-5e56-43cb-b32a-8137a2e0a3ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:35.650300 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-417b0c77-5e56-43cb-b32a-8137a2e0a3ce admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:35.650559 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-417b0c77-5e56-43cb-b32a-8137a2e0a3ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:35.650835 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-417b0c77-5e56-43cb-b32a-8137a2e0a3ce admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:35.651477 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-417b0c77-5e56-43cb-b32a-8137a2e0a3ce admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:35.654009 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-417b0c77-5e56-43cb-b32a-8137a2e0a3ce admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:35.669696 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1032/2087] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:37:35 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:37:35.673948 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd10b68a-b7f3-41d6-aa1d-c570c2b2f96b admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:35.685134 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd10b68a-b7f3-41d6-aa1d-c570c2b2f96b admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:35.693636 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a36669c6-22ba-4c65-ab19-936f2eb349a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:35.694933 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a36669c6-22ba-4c65-ab19-936f2eb349a3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:35.695211 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a36669c6-22ba-4c65-ab19-936f2eb349a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:35.695489 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a36669c6-22ba-4c65-ab19-936f2eb349a3 admin admin] PATH_INFO: `/v3/projects/58129f69bb054a0f939c897a66ce9f07/users/3ee22e595a4e4a0c80026bf8c1430ee3/roles/c51fe7c5ad804040879acfb54269455f` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:35.718616 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a36669c6-22ba-4c65-ab19-936f2eb349a3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=58129f69bb054a0f939c897a66ce9f07, user_id=3ee22e595a4e4a0c80026bf8c1430ee3, role_id=c51fe7c5ad804040879acfb54269455f)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:35.721534 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a36669c6-22ba-4c65-ab19-936f2eb349a3 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:35.754182 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1056/2088] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:37:35 2022] PUT /identity/v3/projects/58129f69bb054a0f939c897a66ce9f07/users/3ee22e595a4e4a0c80026bf8c1430ee3/roles/c51fe7c5ad804040879acfb54269455f => generated 0 bytes in 80 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 04:37:35.757924 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-417b0c77-5e56-43cb-b32a-8137a2e0a3ce admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:35.764449 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:37:35.770562 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-417b0c77-5e56-43cb-b32a-8137a2e0a3ce admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:35.854534 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6543be43-c08e-4cfc-9c1b-a01dabe27eb7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:35.855437 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6543be43-c08e-4cfc-9c1b-a01dabe27eb7 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:35.855851 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6543be43-c08e-4cfc-9c1b-a01dabe27eb7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:35.856775 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6543be43-c08e-4cfc-9c1b-a01dabe27eb7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:35.858085 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6543be43-c08e-4cfc-9c1b-a01dabe27eb7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:35.860252 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6543be43-c08e-4cfc-9c1b-a01dabe27eb7 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:35.886140 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1033/2089] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:37:35 2022] GET /identity/v3/roles => generated 1420 bytes in 125 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 25 04:37:35.887984 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a36669c6-22ba-4c65-ab19-936f2eb349a3 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:35.905950 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a36669c6-22ba-4c65-ab19-936f2eb349a3 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:35.911080 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ffbf74e-ed71-471e-82f0-a52b66b9069c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:35.913413 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ffbf74e-ed71-471e-82f0-a52b66b9069c admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:35.913769 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ffbf74e-ed71-471e-82f0-a52b66b9069c admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:35.914337 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ffbf74e-ed71-471e-82f0-a52b66b9069c admin admin] PATH_INFO: `/v3/projects/58129f69bb054a0f939c897a66ce9f07/users/3ee22e595a4e4a0c80026bf8c1430ee3/roles/c51fe7c5ad804040879acfb54269455f` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:35.916308 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ffbf74e-ed71-471e-82f0-a52b66b9069c admin admin] RBAC: Authorizing `identity:create_grant(project_id=58129f69bb054a0f939c897a66ce9f07, user_id=3ee22e595a4e4a0c80026bf8c1430ee3, role_id=c51fe7c5ad804040879acfb54269455f)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:35.919665 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ffbf74e-ed71-471e-82f0-a52b66b9069c admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:35.935609 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1057/2090] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:37:35 2022] PUT /identity/v3/projects/58129f69bb054a0f939c897a66ce9f07/users/3ee22e595a4e4a0c80026bf8c1430ee3/roles/c51fe7c5ad804040879acfb54269455f => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:37:35.945994 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9985b99e-c45f-48aa-b7a9-d70e04a0d5bc None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:35.946278 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9985b99e-c45f-48aa-b7a9-d70e04a0d5bc None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:35.946552 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9985b99e-c45f-48aa-b7a9-d70e04a0d5bc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:35.988441 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-9985b99e-c45f-48aa-b7a9-d70e04a0d5bc None None] MFA Rules not processed for user `3ee22e595a4e4a0c80026bf8c1430ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:37:35.989916 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-9985b99e-c45f-48aa-b7a9-d70e04a0d5bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:37:36.088670 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1034/2091] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:37:35 2022] POST /identity/v3/auth/tokens => generated 3353 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:37:36.100036 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ffbf74e-ed71-471e-82f0-a52b66b9069c admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:36.102472 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-1ffbf74e-ed71-471e-82f0-a52b66b9069c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:37:36.129182 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9985b99e-c45f-48aa-b7a9-d70e04a0d5bc None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:36.192794 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ffbf74e-ed71-471e-82f0-a52b66b9069c admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:36.197721 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf3d6824-7ae6-4b9b-90d9-cb3b20e8f37f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ffbc136c77b84e8c87128f7247f4ec03', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:36.198358 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf3d6824-7ae6-4b9b-90d9-cb3b20e8f37f service cinder] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:36.198837 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf3d6824-7ae6-4b9b-90d9-cb3b20e8f37f service cinder] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:36.199117 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf3d6824-7ae6-4b9b-90d9-cb3b20e8f37f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:36.201659 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9985b99e-c45f-48aa-b7a9-d70e04a0d5bc None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:36.206312 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00a786eb-9856-45c5-b4ee-171bccd1ecb0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ffbc136c77b84e8c87128f7247f4ec03', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:36.206998 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00a786eb-9856-45c5-b4ee-171bccd1ecb0 service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:36.207484 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00a786eb-9856-45c5-b4ee-171bccd1ecb0 service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:36.207794 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00a786eb-9856-45c5-b4ee-171bccd1ecb0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:36.209796 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-00a786eb-9856-45c5-b4ee-171bccd1ecb0 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:37:36.210604 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf3d6824-7ae6-4b9b-90d9-cb3b20e8f37f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:36.212131 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf3d6824-7ae6-4b9b-90d9-cb3b20e8f37f service cinder] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:36.238308 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1058/2092] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:37:36 2022] GET /identity/v3/auth/tokens => generated 3353 bytes in 128 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:37:36.264492 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:37:36.312816 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00a786eb-9856-45c5-b4ee-171bccd1ecb0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:36.315090 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00a786eb-9856-45c5-b4ee-171bccd1ecb0 service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:36.333798 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1035/2093] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:37:36 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 206 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:37:36.447481 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf3d6824-7ae6-4b9b-90d9-cb3b20e8f37f service cinder] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:36.456919 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf3d6824-7ae6-4b9b-90d9-cb3b20e8f37f service cinder] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:36.461412 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33c21830-1320-4fbd-8d5e-d51088057731 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ffbc136c77b84e8c87128f7247f4ec03', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:36.462171 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33c21830-1320-4fbd-8d5e-d51088057731 service cinder] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:36.462489 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33c21830-1320-4fbd-8d5e-d51088057731 service cinder] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:36.462781 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33c21830-1320-4fbd-8d5e-d51088057731 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:36.494643 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:37:36.536248 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33c21830-1320-4fbd-8d5e-d51088057731 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:36.537912 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33c21830-1320-4fbd-8d5e-d51088057731 service cinder] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:36.548079 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1059/2094] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:37:36 2022] GET /identity/v3/auth/tokens => generated 3284 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 25 04:37:46.324815 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=12 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:37:46.543357 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=18 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:37:47.851268 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00a786eb-9856-45c5-b4ee-171bccd1ecb0 service cinder] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:47.853390 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-00a786eb-9856-45c5-b4ee-171bccd1ecb0 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:37:47.931976 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00a786eb-9856-45c5-b4ee-171bccd1ecb0 service cinder] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:47.937347 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c46ab29-e025-43e8-a3f5-d24d58e09232 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:47.938126 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c46ab29-e025-43e8-a3f5-d24d58e09232 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:47.938509 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c46ab29-e025-43e8-a3f5-d24d58e09232 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:47.938905 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c46ab29-e025-43e8-a3f5-d24d58e09232 admin admin] PATH_INFO: `/v3/users/a5dc5e5e9bb34eb7a3242f169828ddfa` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:47.940786 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c46ab29-e025-43e8-a3f5-d24d58e09232 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a5dc5e5e9bb34eb7a3242f169828ddfa)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:47.943436 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c46ab29-e025-43e8-a3f5-d24d58e09232 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:47.994818 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-7c46ab29-e025-43e8-a3f5-d24d58e09232 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a5dc5e5e9bb34eb7a3242f169828ddfa'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:48.002855 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-7c46ab29-e025-43e8-a3f5-d24d58e09232 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a5dc5e5e9bb34eb7a3242f169828ddfa'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:48.015808 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-7c46ab29-e025-43e8-a3f5-d24d58e09232 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a5dc5e5e9bb34eb7a3242f169828ddfa'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:48.016518 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-7c46ab29-e025-43e8-a3f5-d24d58e09232 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a5dc5e5e9bb34eb7a3242f169828ddfa'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:48.023708 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-7c46ab29-e025-43e8-a3f5-d24d58e09232 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a5dc5e5e9bb34eb7a3242f169828ddfa'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:48.031187 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1036/2095] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:37:47 2022] DELETE /identity/v3/users/a5dc5e5e9bb34eb7a3242f169828ddfa => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:37:48.035860 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a25e59e0-3b63-4653-8221-d061bea41ea6 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:48.036331 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a25e59e0-3b63-4653-8221-d061bea41ea6 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:48.036540 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a25e59e0-3b63-4653-8221-d061bea41ea6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:48.059950 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-a25e59e0-3b63-4653-8221-d061bea41ea6 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:37:48.061220 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-a25e59e0-3b63-4653-8221-d061bea41ea6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:37:48.120622 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1060/2096] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:37:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:37:48.131920 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c46ab29-e025-43e8-a3f5-d24d58e09232 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:48.133065 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-7c46ab29-e025-43e8-a3f5-d24d58e09232 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:37:48.194077 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c46ab29-e025-43e8-a3f5-d24d58e09232 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:48.198937 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5946aa86-9042-40fd-9ef5-21d3c916dc28 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:48.199798 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5946aa86-9042-40fd-9ef5-21d3c916dc28 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:48.200065 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5946aa86-9042-40fd-9ef5-21d3c916dc28 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:48.200337 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5946aa86-9042-40fd-9ef5-21d3c916dc28 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:48.211052 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5946aa86-9042-40fd-9ef5-21d3c916dc28 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:48.212586 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5946aa86-9042-40fd-9ef5-21d3c916dc28 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:48.222950 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1037/2097] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:37:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:37:48.505205 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a25e59e0-3b63-4653-8221-d061bea41ea6 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:48.506940 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-a25e59e0-3b63-4653-8221-d061bea41ea6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:37:48.519701 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a25e59e0-3b63-4653-8221-d061bea41ea6 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:48.524463 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f40a096a-c2a4-42ef-8151-fd3e1ad1ae69 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:48.525138 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f40a096a-c2a4-42ef-8151-fd3e1ad1ae69 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:48.525420 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f40a096a-c2a4-42ef-8151-fd3e1ad1ae69 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:48.525694 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f40a096a-c2a4-42ef-8151-fd3e1ad1ae69 admin admin] PATH_INFO: `/v3/projects/1c7b3c70b5ae40f092a9a009b3161b9b` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:48.527488 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f40a096a-c2a4-42ef-8151-fd3e1ad1ae69 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1c7b3c70b5ae40f092a9a009b3161b9b)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:48.529448 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f40a096a-c2a4-42ef-8151-fd3e1ad1ae69 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:48.580923 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-f40a096a-c2a4-42ef-8151-fd3e1ad1ae69 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1c7b3c70b5ae40f092a9a009b3161b9b', 'request_id': 'req-f40a096a-c2a4-42ef-8151-fd3e1ad1ae69'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:48.589694 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-f40a096a-c2a4-42ef-8151-fd3e1ad1ae69 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1c7b3c70b5ae40f092a9a009b3161b9b', 'request_id': 'req-f40a096a-c2a4-42ef-8151-fd3e1ad1ae69'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:48.595515 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-f40a096a-c2a4-42ef-8151-fd3e1ad1ae69 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1c7b3c70b5ae40f092a9a009b3161b9b', 'request_id': 'req-f40a096a-c2a4-42ef-8151-fd3e1ad1ae69'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:48.629437 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-f40a096a-c2a4-42ef-8151-fd3e1ad1ae69 admin admin] The token cache is being invalidate because project 1c7b3c70b5ae40f092a9a009b3161b9b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81434) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:37:48.629734 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-f40a096a-c2a4-42ef-8151-fd3e1ad1ae69 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:48.630411 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-f40a096a-c2a4-42ef-8151-fd3e1ad1ae69 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:48.631977 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1061/2098] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:37:48 2022] DELETE /identity/v3/projects/1c7b3c70b5ae40f092a9a009b3161b9b => generated 0 bytes in 127 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:37:48.636298 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5946aa86-9042-40fd-9ef5-21d3c916dc28 service neutron] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:48.647751 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5946aa86-9042-40fd-9ef5-21d3c916dc28 service neutron] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:48.668761 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:37:48.717151 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dea0086c-3806-4ccb-833c-5d538c99ffa9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:48.717795 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea0086c-3806-4ccb-833c-5d538c99ffa9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:48.718092 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea0086c-3806-4ccb-833c-5d538c99ffa9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:48.718358 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea0086c-3806-4ccb-833c-5d538c99ffa9 admin admin] PATH_INFO: `/v3/users/d44e6619cda649bf9cdc586139249854` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:48.719501 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dea0086c-3806-4ccb-833c-5d538c99ffa9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d44e6619cda649bf9cdc586139249854)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:48.721460 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dea0086c-3806-4ccb-833c-5d538c99ffa9 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:48.773618 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-dea0086c-3806-4ccb-833c-5d538c99ffa9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd44e6619cda649bf9cdc586139249854'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:48.781899 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-dea0086c-3806-4ccb-833c-5d538c99ffa9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd44e6619cda649bf9cdc586139249854'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:48.795071 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-dea0086c-3806-4ccb-833c-5d538c99ffa9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd44e6619cda649bf9cdc586139249854'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:48.795824 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-dea0086c-3806-4ccb-833c-5d538c99ffa9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd44e6619cda649bf9cdc586139249854'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:48.802644 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-dea0086c-3806-4ccb-833c-5d538c99ffa9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd44e6619cda649bf9cdc586139249854'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:48.809141 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1038/2099] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:37:48 2022] DELETE /identity/v3/users/d44e6619cda649bf9cdc586139249854 => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 04:37:49.094411 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f40a096a-c2a4-42ef-8151-fd3e1ad1ae69 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:49.101440 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-f40a096a-c2a4-42ef-8151-fd3e1ad1ae69 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:37:49.118900 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f40a096a-c2a4-42ef-8151-fd3e1ad1ae69 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:49.199750 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f99cf21-3f93-4925-85ad-fbceeff1e7b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:49.201076 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f99cf21-3f93-4925-85ad-fbceeff1e7b1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:49.201368 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f99cf21-3f93-4925-85ad-fbceeff1e7b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:49.201663 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f99cf21-3f93-4925-85ad-fbceeff1e7b1 admin admin] PATH_INFO: `/v3/projects/01106f395c3f4512959f7f967dcfd82b` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:49.202921 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f99cf21-3f93-4925-85ad-fbceeff1e7b1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=01106f395c3f4512959f7f967dcfd82b)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:49.204969 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f99cf21-3f93-4925-85ad-fbceeff1e7b1 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:49.241454 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:37:49.260346 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-9f99cf21-3f93-4925-85ad-fbceeff1e7b1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '01106f395c3f4512959f7f967dcfd82b', 'request_id': 'req-9f99cf21-3f93-4925-85ad-fbceeff1e7b1'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:49.268932 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-9f99cf21-3f93-4925-85ad-fbceeff1e7b1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '01106f395c3f4512959f7f967dcfd82b', 'request_id': 'req-9f99cf21-3f93-4925-85ad-fbceeff1e7b1'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:49.274404 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-9f99cf21-3f93-4925-85ad-fbceeff1e7b1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '01106f395c3f4512959f7f967dcfd82b', 'request_id': 'req-9f99cf21-3f93-4925-85ad-fbceeff1e7b1'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:49.285259 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-9f99cf21-3f93-4925-85ad-fbceeff1e7b1 admin admin] The token cache is being invalidate because project 01106f395c3f4512959f7f967dcfd82b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81434) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:37:49.285547 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-9f99cf21-3f93-4925-85ad-fbceeff1e7b1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:49.286173 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-9f99cf21-3f93-4925-85ad-fbceeff1e7b1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:49.287668 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1062/2100] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:37:49 2022] DELETE /identity/v3/projects/01106f395c3f4512959f7f967dcfd82b => generated 0 bytes in 194 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:37:49.291798 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dea0086c-3806-4ccb-833c-5d538c99ffa9 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:49.293520 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-dea0086c-3806-4ccb-833c-5d538c99ffa9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:37:49.306174 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dea0086c-3806-4ccb-833c-5d538c99ffa9 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:49.377360 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-658d8aac-12a4-47b0-b68a-267c927d6343 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:49.378076 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-658d8aac-12a4-47b0-b68a-267c927d6343 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:49.380921 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-658d8aac-12a4-47b0-b68a-267c927d6343 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:49.381292 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-658d8aac-12a4-47b0-b68a-267c927d6343 admin admin] PATH_INFO: `/v3/users/8810f783fc55406f9bf2ca0b0bcd6d1b` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:49.382558 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-658d8aac-12a4-47b0-b68a-267c927d6343 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8810f783fc55406f9bf2ca0b0bcd6d1b)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:49.384567 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-658d8aac-12a4-47b0-b68a-267c927d6343 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:49.431796 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=83,keystone:DELETE=16,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:37:49.435507 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-658d8aac-12a4-47b0-b68a-267c927d6343 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8810f783fc55406f9bf2ca0b0bcd6d1b'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:49.444336 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-658d8aac-12a4-47b0-b68a-267c927d6343 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8810f783fc55406f9bf2ca0b0bcd6d1b'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:49.458923 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-658d8aac-12a4-47b0-b68a-267c927d6343 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8810f783fc55406f9bf2ca0b0bcd6d1b'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:49.460173 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-658d8aac-12a4-47b0-b68a-267c927d6343 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8810f783fc55406f9bf2ca0b0bcd6d1b'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:49.474042 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-658d8aac-12a4-47b0-b68a-267c927d6343 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8810f783fc55406f9bf2ca0b0bcd6d1b'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:49.484333 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1039/2101] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:37:49 2022] DELETE /identity/v3/users/8810f783fc55406f9bf2ca0b0bcd6d1b => generated 0 bytes in 193 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:37:49.841662 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f99cf21-3f93-4925-85ad-fbceeff1e7b1 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:49.843485 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-9f99cf21-3f93-4925-85ad-fbceeff1e7b1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:37:49.859892 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f99cf21-3f93-4925-85ad-fbceeff1e7b1 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:49.953462 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac5df09d-f6c3-4101-ab40-b7c91b31597f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:49.954115 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac5df09d-f6c3-4101-ab40-b7c91b31597f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:49.954402 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac5df09d-f6c3-4101-ab40-b7c91b31597f admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:49.954664 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac5df09d-f6c3-4101-ab40-b7c91b31597f admin admin] PATH_INFO: `/v3/projects/fa429cb2cdd2454291b51b175d1b15fe` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:49.955791 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac5df09d-f6c3-4101-ab40-b7c91b31597f admin admin] RBAC: Authorizing `identity:delete_project(project_id=fa429cb2cdd2454291b51b175d1b15fe)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:49.957797 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac5df09d-f6c3-4101-ab40-b7c91b31597f admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:50.009996 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-ac5df09d-f6c3-4101-ab40-b7c91b31597f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fa429cb2cdd2454291b51b175d1b15fe', 'request_id': 'req-ac5df09d-f6c3-4101-ab40-b7c91b31597f'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:50.019134 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-ac5df09d-f6c3-4101-ab40-b7c91b31597f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fa429cb2cdd2454291b51b175d1b15fe', 'request_id': 'req-ac5df09d-f6c3-4101-ab40-b7c91b31597f'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:50.026020 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-ac5df09d-f6c3-4101-ab40-b7c91b31597f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fa429cb2cdd2454291b51b175d1b15fe', 'request_id': 'req-ac5df09d-f6c3-4101-ab40-b7c91b31597f'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:50.037561 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-ac5df09d-f6c3-4101-ab40-b7c91b31597f admin admin] The token cache is being invalidate because project fa429cb2cdd2454291b51b175d1b15fe was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81434) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:37:50.037857 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-ac5df09d-f6c3-4101-ab40-b7c91b31597f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:50.038518 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-ac5df09d-f6c3-4101-ab40-b7c91b31597f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:50.040018 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1063/2102] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:37:49 2022] DELETE /identity/v3/projects/fa429cb2cdd2454291b51b175d1b15fe => generated 0 bytes in 200 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 04:37:50.443213 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-658d8aac-12a4-47b0-b68a-267c927d6343 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:50.444943 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-658d8aac-12a4-47b0-b68a-267c927d6343 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:37:50.460383 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-658d8aac-12a4-47b0-b68a-267c927d6343 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:50.520837 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7733606-80f3-41c2-a846-438993c076fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:50.521369 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7733606-80f3-41c2-a846-438993c076fa admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:50.521672 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7733606-80f3-41c2-a846-438993c076fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:50.521946 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7733606-80f3-41c2-a846-438993c076fa admin admin] PATH_INFO: `/v3/users/71461b6fab73487b84d40cd31e3c8b9d` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:50.523063 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7733606-80f3-41c2-a846-438993c076fa admin admin] RBAC: Authorizing `identity:delete_user(user_id=71461b6fab73487b84d40cd31e3c8b9d)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:50.525040 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7733606-80f3-41c2-a846-438993c076fa admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:50.572813 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-a7733606-80f3-41c2-a846-438993c076fa admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '71461b6fab73487b84d40cd31e3c8b9d'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:50.581952 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-a7733606-80f3-41c2-a846-438993c076fa admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '71461b6fab73487b84d40cd31e3c8b9d'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:50.594650 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-a7733606-80f3-41c2-a846-438993c076fa admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '71461b6fab73487b84d40cd31e3c8b9d'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:50.595291 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-a7733606-80f3-41c2-a846-438993c076fa admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '71461b6fab73487b84d40cd31e3c8b9d'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:50.602118 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-a7733606-80f3-41c2-a846-438993c076fa admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '71461b6fab73487b84d40cd31e3c8b9d'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:50.608835 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1040/2103] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:37:50 2022] DELETE /identity/v3/users/71461b6fab73487b84d40cd31e3c8b9d => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:37:50.613256 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e7b026f-dbec-4c0b-9eb3-7100597e0c98 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:50.613699 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e7b026f-dbec-4c0b-9eb3-7100597e0c98 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:50.613789 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e7b026f-dbec-4c0b-9eb3-7100597e0c98 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:50.633120 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-9e7b026f-dbec-4c0b-9eb3-7100597e0c98 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:37:50.634328 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-9e7b026f-dbec-4c0b-9eb3-7100597e0c98 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:37:50.675079 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:DELETE=13,keystone:SELECT=85,keystone:INSERT=2 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:37:50.694869 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1064/2104] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:37:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:37:50.704417 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7733606-80f3-41c2-a846-438993c076fa admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:50.706092 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-a7733606-80f3-41c2-a846-438993c076fa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:37:50.751919 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:DELETE=14,keystone:SELECT=84,keystone:INSERT=2 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:37:50.766981 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7733606-80f3-41c2-a846-438993c076fa admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:50.770680 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15ce5ad8-7a6e-4338-a847-fe9eba13aedc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:50.771334 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15ce5ad8-7a6e-4338-a847-fe9eba13aedc service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:50.771563 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15ce5ad8-7a6e-4338-a847-fe9eba13aedc service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:50.771837 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15ce5ad8-7a6e-4338-a847-fe9eba13aedc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:50.782319 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15ce5ad8-7a6e-4338-a847-fe9eba13aedc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:50.783996 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15ce5ad8-7a6e-4338-a847-fe9eba13aedc service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:50.794259 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1041/2105] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:37:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:37:51.058316 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e7b026f-dbec-4c0b-9eb3-7100597e0c98 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:51.059865 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-9e7b026f-dbec-4c0b-9eb3-7100597e0c98 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:37:51.072875 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e7b026f-dbec-4c0b-9eb3-7100597e0c98 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:51.077821 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d6751e1-177a-460a-8792-2d7de5de40b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:51.078857 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d6751e1-177a-460a-8792-2d7de5de40b1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:51.078857 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d6751e1-177a-460a-8792-2d7de5de40b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:51.079073 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d6751e1-177a-460a-8792-2d7de5de40b1 admin admin] PATH_INFO: `/v3/projects/8c82c76db2be48ea95bd8e55de046a95` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:51.080137 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d6751e1-177a-460a-8792-2d7de5de40b1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8c82c76db2be48ea95bd8e55de046a95)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:51.082094 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d6751e1-177a-460a-8792-2d7de5de40b1 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:51.130184 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-1d6751e1-177a-460a-8792-2d7de5de40b1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8c82c76db2be48ea95bd8e55de046a95', 'request_id': 'req-1d6751e1-177a-460a-8792-2d7de5de40b1'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:51.137459 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-1d6751e1-177a-460a-8792-2d7de5de40b1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8c82c76db2be48ea95bd8e55de046a95', 'request_id': 'req-1d6751e1-177a-460a-8792-2d7de5de40b1'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:51.142982 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-1d6751e1-177a-460a-8792-2d7de5de40b1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8c82c76db2be48ea95bd8e55de046a95', 'request_id': 'req-1d6751e1-177a-460a-8792-2d7de5de40b1'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:51.154035 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-1d6751e1-177a-460a-8792-2d7de5de40b1 admin admin] The token cache is being invalidate because project 8c82c76db2be48ea95bd8e55de046a95 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81434) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:37:51.154320 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-1d6751e1-177a-460a-8792-2d7de5de40b1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:51.154949 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-1d6751e1-177a-460a-8792-2d7de5de40b1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:51.156437 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1065/2106] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:37:51 2022] DELETE /identity/v3/projects/8c82c76db2be48ea95bd8e55de046a95 => generated 0 bytes in 99 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:37:51.160572 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15ce5ad8-7a6e-4338-a847-fe9eba13aedc service neutron] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:51.171169 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15ce5ad8-7a6e-4338-a847-fe9eba13aedc service neutron] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:51.229267 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76bef0dc-84d8-4c79-8606-56c81cfea0df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:51.229959 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76bef0dc-84d8-4c79-8606-56c81cfea0df admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:51.230240 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76bef0dc-84d8-4c79-8606-56c81cfea0df admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:51.230536 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76bef0dc-84d8-4c79-8606-56c81cfea0df admin admin] PATH_INFO: `/v3/users/3ee22e595a4e4a0c80026bf8c1430ee3` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:51.231665 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76bef0dc-84d8-4c79-8606-56c81cfea0df admin admin] RBAC: Authorizing `identity:delete_user(user_id=3ee22e595a4e4a0c80026bf8c1430ee3)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:51.233623 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76bef0dc-84d8-4c79-8606-56c81cfea0df admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:51.284689 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-76bef0dc-84d8-4c79-8606-56c81cfea0df admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3ee22e595a4e4a0c80026bf8c1430ee3'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:51.292167 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-76bef0dc-84d8-4c79-8606-56c81cfea0df admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3ee22e595a4e4a0c80026bf8c1430ee3'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:51.304032 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-76bef0dc-84d8-4c79-8606-56c81cfea0df admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3ee22e595a4e4a0c80026bf8c1430ee3'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:51.304596 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-76bef0dc-84d8-4c79-8606-56c81cfea0df admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3ee22e595a4e4a0c80026bf8c1430ee3'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:51.311587 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-76bef0dc-84d8-4c79-8606-56c81cfea0df admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3ee22e595a4e4a0c80026bf8c1430ee3'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:51.317264 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1042/2107] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:37:51 2022] DELETE /identity/v3/users/3ee22e595a4e4a0c80026bf8c1430ee3 => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 04:37:51.802473 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d6751e1-177a-460a-8792-2d7de5de40b1 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:51.803988 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-1d6751e1-177a-460a-8792-2d7de5de40b1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:37:51.818332 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d6751e1-177a-460a-8792-2d7de5de40b1 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:51.879545 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fca06fa-6485-430e-b8f2-0e6c35e96394 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:51.880109 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fca06fa-6485-430e-b8f2-0e6c35e96394 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:51.880389 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fca06fa-6485-430e-b8f2-0e6c35e96394 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:51.880691 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fca06fa-6485-430e-b8f2-0e6c35e96394 admin admin] PATH_INFO: `/v3/projects/58129f69bb054a0f939c897a66ce9f07` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:51.881799 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fca06fa-6485-430e-b8f2-0e6c35e96394 admin admin] RBAC: Authorizing `identity:delete_project(project_id=58129f69bb054a0f939c897a66ce9f07)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:51.883738 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fca06fa-6485-430e-b8f2-0e6c35e96394 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:51.932439 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-5fca06fa-6485-430e-b8f2-0e6c35e96394 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '58129f69bb054a0f939c897a66ce9f07', 'request_id': 'req-5fca06fa-6485-430e-b8f2-0e6c35e96394'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:51.940178 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-5fca06fa-6485-430e-b8f2-0e6c35e96394 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '58129f69bb054a0f939c897a66ce9f07', 'request_id': 'req-5fca06fa-6485-430e-b8f2-0e6c35e96394'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:51.947663 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-5fca06fa-6485-430e-b8f2-0e6c35e96394 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '58129f69bb054a0f939c897a66ce9f07', 'request_id': 'req-5fca06fa-6485-430e-b8f2-0e6c35e96394'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:51.957146 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:37:51.962477 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-5fca06fa-6485-430e-b8f2-0e6c35e96394 admin admin] The token cache is being invalidate because project 58129f69bb054a0f939c897a66ce9f07 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81434) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:37:51.962936 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-5fca06fa-6485-430e-b8f2-0e6c35e96394 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:51.964108 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-5fca06fa-6485-430e-b8f2-0e6c35e96394 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:51.965933 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1066/2108] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:37:51 2022] DELETE /identity/v3/projects/58129f69bb054a0f939c897a66ce9f07 => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:37:51.970756 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e06750b4-1caf-46c5-b7ae-ca309d595c94 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:51.971244 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e06750b4-1caf-46c5-b7ae-ca309d595c94 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:51.971336 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e06750b4-1caf-46c5-b7ae-ca309d595c94 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:51.991404 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-e06750b4-1caf-46c5-b7ae-ca309d595c94 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:37:51.992611 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-e06750b4-1caf-46c5-b7ae-ca309d595c94 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:37:52.050925 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1043/2109] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:37:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:37:52.060698 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6da2776-3ee6-4534-8a30-4c3191b0bf12 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:52.061033 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6da2776-3ee6-4534-8a30-4c3191b0bf12 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:52.061306 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6da2776-3ee6-4534-8a30-4c3191b0bf12 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:52.081242 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-a6da2776-3ee6-4534-8a30-4c3191b0bf12 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:37:52.082471 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-a6da2776-3ee6-4534-8a30-4c3191b0bf12 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:37:52.091441 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1067/2110] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:37:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:37:52.095675 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e06750b4-1caf-46c5-b7ae-ca309d595c94 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:52.100511 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:37:52.107800 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e06750b4-1caf-46c5-b7ae-ca309d595c94 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:52.113810 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ca8cd3f-c2fa-43aa-b09e-1a40b7155a0a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:52.114405 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca8cd3f-c2fa-43aa-b09e-1a40b7155a0a admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:52.114681 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca8cd3f-c2fa-43aa-b09e-1a40b7155a0a admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:52.114951 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca8cd3f-c2fa-43aa-b09e-1a40b7155a0a admin admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:52.115587 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ca8cd3f-c2fa-43aa-b09e-1a40b7155a0a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:37:52.115889 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ca8cd3f-c2fa-43aa-b09e-1a40b7155a0a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:52.117516 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ca8cd3f-c2fa-43aa-b09e-1a40b7155a0a admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:52.129855 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1044/2111] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Sep 25 04:37:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:37:52.134120 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6da2776-3ee6-4534-8a30-4c3191b0bf12 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:52.143392 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6da2776-3ee6-4534-8a30-4c3191b0bf12 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:52.148061 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13c5b6b9-ff00-4521-8f90-5c0ec1101612 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:52.148678 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13c5b6b9-ff00-4521-8f90-5c0ec1101612 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:52.148960 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13c5b6b9-ff00-4521-8f90-5c0ec1101612 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:52.149228 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13c5b6b9-ff00-4521-8f90-5c0ec1101612 admin admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:52.149849 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13c5b6b9-ff00-4521-8f90-5c0ec1101612 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:52.152230 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13c5b6b9-ff00-4521-8f90-5c0ec1101612 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:52.180334 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1068/2112] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:37:52 2022] POST /identity/v3/projects => generated 373 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:37:52.184580 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ca8cd3f-c2fa-43aa-b09e-1a40b7155a0a admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:52.194392 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ca8cd3f-c2fa-43aa-b09e-1a40b7155a0a admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:52.249921 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d45d363f-150c-4fa1-b7dc-266ad44ed965 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:52.250518 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d45d363f-150c-4fa1-b7dc-266ad44ed965 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:52.250794 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d45d363f-150c-4fa1-b7dc-266ad44ed965 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:52.251064 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d45d363f-150c-4fa1-b7dc-266ad44ed965 admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:52.251693 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d45d363f-150c-4fa1-b7dc-266ad44ed965 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:52.253756 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d45d363f-150c-4fa1-b7dc-266ad44ed965 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:52.275473 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1045/2113] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:37:52 2022] POST /identity/v3/users => generated 287 bytes in 91 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:37:52.280214 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13c5b6b9-ff00-4521-8f90-5c0ec1101612 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:52.290861 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13c5b6b9-ff00-4521-8f90-5c0ec1101612 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:52.296333 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64cd6cd3-cb51-4c33-8a62-6b24e8f4247e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:52.297012 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64cd6cd3-cb51-4c33-8a62-6b24e8f4247e admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:52.297333 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64cd6cd3-cb51-4c33-8a62-6b24e8f4247e admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:52.297613 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64cd6cd3-cb51-4c33-8a62-6b24e8f4247e admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:52.298262 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64cd6cd3-cb51-4c33-8a62-6b24e8f4247e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:52.299895 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64cd6cd3-cb51-4c33-8a62-6b24e8f4247e admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:52.310513 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1069/2114] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:37:52 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:37:52.314711 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d45d363f-150c-4fa1-b7dc-266ad44ed965 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:52.324956 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d45d363f-150c-4fa1-b7dc-266ad44ed965 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:52.329710 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b2a6532-d69c-4ee6-9c5e-aadd00b180ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:52.330313 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b2a6532-d69c-4ee6-9c5e-aadd00b180ff admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:52.330600 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b2a6532-d69c-4ee6-9c5e-aadd00b180ff admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:52.330877 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b2a6532-d69c-4ee6-9c5e-aadd00b180ff admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:52.331520 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b2a6532-d69c-4ee6-9c5e-aadd00b180ff admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:52.333422 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b2a6532-d69c-4ee6-9c5e-aadd00b180ff admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:52.345997 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1046/2115] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:37:52 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:37:52.350263 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64cd6cd3-cb51-4c33-8a62-6b24e8f4247e admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:52.360988 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64cd6cd3-cb51-4c33-8a62-6b24e8f4247e admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:52.366445 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c71bc499-45ce-46eb-8ffc-c3e9db1b285d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:52.367182 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c71bc499-45ce-46eb-8ffc-c3e9db1b285d admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:52.367459 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c71bc499-45ce-46eb-8ffc-c3e9db1b285d admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:52.367737 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c71bc499-45ce-46eb-8ffc-c3e9db1b285d admin admin] PATH_INFO: `/v3/projects/55603a2f8726405d949011941333231b/users/c9879f9468d64eeca3e2d096987863a7/roles/8f6cc574056c4278bc9778f14dc4b411` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:52.383302 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c71bc499-45ce-46eb-8ffc-c3e9db1b285d admin admin] RBAC: Authorizing `identity:create_grant(project_id=55603a2f8726405d949011941333231b, user_id=c9879f9468d64eeca3e2d096987863a7, role_id=8f6cc574056c4278bc9778f14dc4b411)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:52.386528 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c71bc499-45ce-46eb-8ffc-c3e9db1b285d admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:52.400195 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1070/2116] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:37:52 2022] PUT /identity/v3/projects/55603a2f8726405d949011941333231b/users/c9879f9468d64eeca3e2d096987863a7/roles/8f6cc574056c4278bc9778f14dc4b411 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:37:52.411396 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4b62e8b-3ffb-4fac-90bb-145e36fa3202 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:52.411757 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4b62e8b-3ffb-4fac-90bb-145e36fa3202 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:52.412104 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4b62e8b-3ffb-4fac-90bb-145e36fa3202 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:52.441379 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-d4b62e8b-3ffb-4fac-90bb-145e36fa3202 None None] MFA Rules not processed for user `c9879f9468d64eeca3e2d096987863a7`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:37:52.443013 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-d4b62e8b-3ffb-4fac-90bb-145e36fa3202 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:37:52.520878 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1047/2117] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:37:52 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:37:52.535151 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c71bc499-45ce-46eb-8ffc-c3e9db1b285d admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:52.537588 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-c71bc499-45ce-46eb-8ffc-c3e9db1b285d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:37:52.632000 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c71bc499-45ce-46eb-8ffc-c3e9db1b285d admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:52.646777 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-911b8126-1159-4648-8d2a-dec6cae6b129 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ffbc136c77b84e8c87128f7247f4ec03', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:52.647688 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-911b8126-1159-4648-8d2a-dec6cae6b129 service cinder] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:52.648322 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-911b8126-1159-4648-8d2a-dec6cae6b129 service cinder] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:52.648809 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-911b8126-1159-4648-8d2a-dec6cae6b129 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:52.661889 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-911b8126-1159-4648-8d2a-dec6cae6b129 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:52.663502 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-911b8126-1159-4648-8d2a-dec6cae6b129 service cinder] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:52.681457 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1071/2118] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:37:52 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 147 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:37:52.716537 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4b62e8b-3ffb-4fac-90bb-145e36fa3202 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:52.782612 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:37:52.788084 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4b62e8b-3ffb-4fac-90bb-145e36fa3202 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:52.795458 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a5728b6-9767-4c80-a54d-55a2fbeaf962 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:52.796897 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a5728b6-9767-4c80-a54d-55a2fbeaf962 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:52.796897 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a5728b6-9767-4c80-a54d-55a2fbeaf962 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:52.796897 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a5728b6-9767-4c80-a54d-55a2fbeaf962 admin admin] PATH_INFO: `/v3/users/c9879f9468d64eeca3e2d096987863a7` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:52.798387 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a5728b6-9767-4c80-a54d-55a2fbeaf962 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c9879f9468d64eeca3e2d096987863a7)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:52.800525 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a5728b6-9767-4c80-a54d-55a2fbeaf962 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:52.855772 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-8a5728b6-9767-4c80-a54d-55a2fbeaf962 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c9879f9468d64eeca3e2d096987863a7'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:52.869175 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-8a5728b6-9767-4c80-a54d-55a2fbeaf962 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c9879f9468d64eeca3e2d096987863a7'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:52.884266 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-8a5728b6-9767-4c80-a54d-55a2fbeaf962 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c9879f9468d64eeca3e2d096987863a7'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:52.885065 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-8a5728b6-9767-4c80-a54d-55a2fbeaf962 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c9879f9468d64eeca3e2d096987863a7'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:52.896266 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-8a5728b6-9767-4c80-a54d-55a2fbeaf962 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c9879f9468d64eeca3e2d096987863a7'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:52.903204 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1048/2119] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:37:52 2022] DELETE /identity/v3/users/c9879f9468d64eeca3e2d096987863a7 => generated 0 bytes in 187 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:37:52.907678 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88674416-f574-4a59-854b-f64985f81ffc None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:52.908184 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88674416-f574-4a59-854b-f64985f81ffc None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:52.908734 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88674416-f574-4a59-854b-f64985f81ffc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:52.933669 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-88674416-f574-4a59-854b-f64985f81ffc None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:37:52.935058 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-88674416-f574-4a59-854b-f64985f81ffc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:37:53.016409 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1072/2120] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:37:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:37:53.035043 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a5728b6-9767-4c80-a54d-55a2fbeaf962 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:53.037084 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-8a5728b6-9767-4c80-a54d-55a2fbeaf962 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:37:53.100790 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a5728b6-9767-4c80-a54d-55a2fbeaf962 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:53.104486 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8b2bc0b-42ed-45db-955d-60cd73721770 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:53.105444 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b2bc0b-42ed-45db-955d-60cd73721770 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:53.105778 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b2bc0b-42ed-45db-955d-60cd73721770 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:53.106053 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b2bc0b-42ed-45db-955d-60cd73721770 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:53.116173 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8b2bc0b-42ed-45db-955d-60cd73721770 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:53.119844 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8b2bc0b-42ed-45db-955d-60cd73721770 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:53.131506 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1049/2121] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:37:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:37:53.618598 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88674416-f574-4a59-854b-f64985f81ffc None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:53.628101 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-88674416-f574-4a59-854b-f64985f81ffc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:37:53.647240 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88674416-f574-4a59-854b-f64985f81ffc None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:53.652929 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-828a8c69-2439-4dcd-84e3-8889a03014d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:53.653712 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-828a8c69-2439-4dcd-84e3-8889a03014d2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:53.654145 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-828a8c69-2439-4dcd-84e3-8889a03014d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:53.654640 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-828a8c69-2439-4dcd-84e3-8889a03014d2 admin admin] PATH_INFO: `/v3/projects/55603a2f8726405d949011941333231b` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:53.655788 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-828a8c69-2439-4dcd-84e3-8889a03014d2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=55603a2f8726405d949011941333231b)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:53.657854 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-828a8c69-2439-4dcd-84e3-8889a03014d2 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:53.693386 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=96,keystone:INSERT=2,keystone:DELETE=2 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:37:53.707871 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-828a8c69-2439-4dcd-84e3-8889a03014d2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '55603a2f8726405d949011941333231b', 'request_id': 'req-828a8c69-2439-4dcd-84e3-8889a03014d2'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:53.717063 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-828a8c69-2439-4dcd-84e3-8889a03014d2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '55603a2f8726405d949011941333231b', 'request_id': 'req-828a8c69-2439-4dcd-84e3-8889a03014d2'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:53.722957 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-828a8c69-2439-4dcd-84e3-8889a03014d2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '55603a2f8726405d949011941333231b', 'request_id': 'req-828a8c69-2439-4dcd-84e3-8889a03014d2'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:53.734302 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-828a8c69-2439-4dcd-84e3-8889a03014d2 admin admin] The token cache is being invalidate because project 55603a2f8726405d949011941333231b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81434) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:37:53.734602 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-828a8c69-2439-4dcd-84e3-8889a03014d2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:53.735263 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-828a8c69-2439-4dcd-84e3-8889a03014d2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:53.736801 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1073/2122] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:37:53 2022] DELETE /identity/v3/projects/55603a2f8726405d949011941333231b => generated 0 bytes in 119 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:37:53.741605 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0f4766b-521c-44e3-aa7d-e88faca0ff7c None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:53.741913 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0f4766b-521c-44e3-aa7d-e88faca0ff7c None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:53.742200 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0f4766b-521c-44e3-aa7d-e88faca0ff7c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:53.763268 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-e0f4766b-521c-44e3-aa7d-e88faca0ff7c None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:37:53.764481 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-e0f4766b-521c-44e3-aa7d-e88faca0ff7c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:37:53.827045 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1050/2123] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:37:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:37:53.835057 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c41cf6c-58f6-4046-9da7-2eca01e201a5 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:53.835237 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c41cf6c-58f6-4046-9da7-2eca01e201a5 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:53.835578 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c41cf6c-58f6-4046-9da7-2eca01e201a5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:53.859992 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-5c41cf6c-58f6-4046-9da7-2eca01e201a5 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:37:53.861382 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-5c41cf6c-58f6-4046-9da7-2eca01e201a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:37:53.871034 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1074/2124] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:37:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:37:53.878340 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0f4766b-521c-44e3-aa7d-e88faca0ff7c None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:53.888887 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0f4766b-521c-44e3-aa7d-e88faca0ff7c None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:53.896130 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1afb343a-a275-44fd-baf0-2af44459b510 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:53.896574 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1afb343a-a275-44fd-baf0-2af44459b510 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:53.896906 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1afb343a-a275-44fd-baf0-2af44459b510 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:53.897182 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1afb343a-a275-44fd-baf0-2af44459b510 admin admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:53.897845 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1afb343a-a275-44fd-baf0-2af44459b510 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:37:53.898156 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1afb343a-a275-44fd-baf0-2af44459b510 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:53.899851 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1afb343a-a275-44fd-baf0-2af44459b510 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:53.916592 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1051/2125] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Sep 25 04:37:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:37:53.918246 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c41cf6c-58f6-4046-9da7-2eca01e201a5 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:53.927855 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c41cf6c-58f6-4046-9da7-2eca01e201a5 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:53.933639 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb209e9a-b2c2-4e1e-8a08-da10b726d42a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:53.934842 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb209e9a-b2c2-4e1e-8a08-da10b726d42a admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:53.934842 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb209e9a-b2c2-4e1e-8a08-da10b726d42a admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:53.935078 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb209e9a-b2c2-4e1e-8a08-da10b726d42a admin admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:53.935564 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb209e9a-b2c2-4e1e-8a08-da10b726d42a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:53.937499 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb209e9a-b2c2-4e1e-8a08-da10b726d42a admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:53.972716 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1075/2126] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:37:53 2022] POST /identity/v3/projects => generated 377 bytes in 55 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:37:53.976514 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1afb343a-a275-44fd-baf0-2af44459b510 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:53.990778 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1afb343a-a275-44fd-baf0-2af44459b510 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:53.999167 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:37:54.077461 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f39c8620-471f-4dc9-b82b-48de7fa71b4e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:54.078180 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f39c8620-471f-4dc9-b82b-48de7fa71b4e admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:54.078399 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f39c8620-471f-4dc9-b82b-48de7fa71b4e admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:54.078632 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f39c8620-471f-4dc9-b82b-48de7fa71b4e admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:54.079272 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f39c8620-471f-4dc9-b82b-48de7fa71b4e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:54.081254 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f39c8620-471f-4dc9-b82b-48de7fa71b4e admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:54.104000 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1052/2127] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:37:53 2022] POST /identity/v3/users => generated 289 bytes in 128 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:37:54.108705 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb209e9a-b2c2-4e1e-8a08-da10b726d42a admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:54.122081 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb209e9a-b2c2-4e1e-8a08-da10b726d42a admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:54.128469 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-966e139b-ec3b-448f-ab7a-bfe1461daf05 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:54.128822 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-966e139b-ec3b-448f-ab7a-bfe1461daf05 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:54.129117 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-966e139b-ec3b-448f-ab7a-bfe1461daf05 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:54.129388 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-966e139b-ec3b-448f-ab7a-bfe1461daf05 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:54.130038 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-966e139b-ec3b-448f-ab7a-bfe1461daf05 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:54.131777 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-966e139b-ec3b-448f-ab7a-bfe1461daf05 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:54.142573 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1076/2128] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:37:54 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:37:54.146612 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f39c8620-471f-4dc9-b82b-48de7fa71b4e admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:54.156936 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f39c8620-471f-4dc9-b82b-48de7fa71b4e admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:54.162297 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9bbfb31-78e5-4b28-8652-fc0ab9745968 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:54.162918 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9bbfb31-78e5-4b28-8652-fc0ab9745968 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:54.163202 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9bbfb31-78e5-4b28-8652-fc0ab9745968 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:54.163478 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9bbfb31-78e5-4b28-8652-fc0ab9745968 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:54.164127 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9bbfb31-78e5-4b28-8652-fc0ab9745968 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:54.165765 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9bbfb31-78e5-4b28-8652-fc0ab9745968 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:54.183884 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1053/2129] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:37:54 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:37:54.184047 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-966e139b-ec3b-448f-ab7a-bfe1461daf05 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:54.192253 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-966e139b-ec3b-448f-ab7a-bfe1461daf05 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:54.197017 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-939ea0c0-e702-4bef-ba2c-eec82baa8714 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:54.197671 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-939ea0c0-e702-4bef-ba2c-eec82baa8714 admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:54.199245 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-939ea0c0-e702-4bef-ba2c-eec82baa8714 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:54.199245 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-939ea0c0-e702-4bef-ba2c-eec82baa8714 admin admin] PATH_INFO: `/v3/projects/099c5ea5a055473186e6e9e76edb9d3e/users/08499712078f400a8bca5da2742137ec/roles/8f6cc574056c4278bc9778f14dc4b411` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:54.213253 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-939ea0c0-e702-4bef-ba2c-eec82baa8714 admin admin] RBAC: Authorizing `identity:create_grant(project_id=099c5ea5a055473186e6e9e76edb9d3e, user_id=08499712078f400a8bca5da2742137ec, role_id=8f6cc574056c4278bc9778f14dc4b411)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:54.215942 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-939ea0c0-e702-4bef-ba2c-eec82baa8714 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:54.230339 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1077/2130] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:37:54 2022] PUT /identity/v3/projects/099c5ea5a055473186e6e9e76edb9d3e/users/08499712078f400a8bca5da2742137ec/roles/8f6cc574056c4278bc9778f14dc4b411 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 04:37:54.241379 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-215f5fcc-c02e-4ad7-8247-6c38c818306e None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:54.241666 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-215f5fcc-c02e-4ad7-8247-6c38c818306e None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:54.241940 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-215f5fcc-c02e-4ad7-8247-6c38c818306e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:54.271203 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-215f5fcc-c02e-4ad7-8247-6c38c818306e None None] MFA Rules not processed for user `08499712078f400a8bca5da2742137ec`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:37:54.272438 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-215f5fcc-c02e-4ad7-8247-6c38c818306e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:37:54.338487 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1054/2131] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:37:54 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:37:54.350296 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-939ea0c0-e702-4bef-ba2c-eec82baa8714 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:54.352099 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-939ea0c0-e702-4bef-ba2c-eec82baa8714 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:37:54.431551 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-939ea0c0-e702-4bef-ba2c-eec82baa8714 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:54.436112 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-480fded5-4b66-4363-9536-b1786375270a service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '316eaa67d5464091af47efd489b96b87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:54.436757 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-480fded5-4b66-4363-9536-b1786375270a service glance] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:54.437035 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-480fded5-4b66-4363-9536-b1786375270a service glance] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:54.437305 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-480fded5-4b66-4363-9536-b1786375270a service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:54.447335 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-480fded5-4b66-4363-9536-b1786375270a service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:54.449261 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-480fded5-4b66-4363-9536-b1786375270a service glance] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:54.459034 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1078/2132] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:37:54 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 25 04:37:56.524840 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-215f5fcc-c02e-4ad7-8247-6c38c818306e None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:56.526549 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-215f5fcc-c02e-4ad7-8247-6c38c818306e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:37:56.600587 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-215f5fcc-c02e-4ad7-8247-6c38c818306e None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:56.605790 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27da78fb-ddbc-49c8-8075-0fd4f8d05721 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:56.606511 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27da78fb-ddbc-49c8-8075-0fd4f8d05721 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:56.606866 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27da78fb-ddbc-49c8-8075-0fd4f8d05721 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:56.607210 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27da78fb-ddbc-49c8-8075-0fd4f8d05721 admin admin] PATH_INFO: `/v3/users/1d1bf78eb0a94bab884b1df13c081096` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:56.608411 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27da78fb-ddbc-49c8-8075-0fd4f8d05721 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1d1bf78eb0a94bab884b1df13c081096)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:56.610434 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27da78fb-ddbc-49c8-8075-0fd4f8d05721 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:56.626722 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=96,keystone:INSERT=3,keystone:DELETE=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:37:56.660688 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-27da78fb-ddbc-49c8-8075-0fd4f8d05721 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1d1bf78eb0a94bab884b1df13c081096'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:56.670661 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-27da78fb-ddbc-49c8-8075-0fd4f8d05721 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1d1bf78eb0a94bab884b1df13c081096'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:56.684620 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-27da78fb-ddbc-49c8-8075-0fd4f8d05721 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1d1bf78eb0a94bab884b1df13c081096'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:56.685354 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-27da78fb-ddbc-49c8-8075-0fd4f8d05721 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1d1bf78eb0a94bab884b1df13c081096'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:56.692703 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-27da78fb-ddbc-49c8-8075-0fd4f8d05721 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1d1bf78eb0a94bab884b1df13c081096'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:56.699304 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1055/2133] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:37:56 2022] DELETE /identity/v3/users/1d1bf78eb0a94bab884b1df13c081096 => generated 0 bytes in 175 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:37:56.896042 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-480fded5-4b66-4363-9536-b1786375270a service glance] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:56.907784 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-480fded5-4b66-4363-9536-b1786375270a service glance] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:56.965154 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1bb8a3a-39f4-499d-ad9b-4d518b6936bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:56.965787 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1bb8a3a-39f4-499d-ad9b-4d518b6936bb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:56.966097 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1bb8a3a-39f4-499d-ad9b-4d518b6936bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:56.966336 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1bb8a3a-39f4-499d-ad9b-4d518b6936bb admin admin] PATH_INFO: `/v3/projects/80e5b6ce14ab47dfac99cc80153d7735` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:56.967445 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1bb8a3a-39f4-499d-ad9b-4d518b6936bb admin admin] RBAC: Authorizing `identity:delete_project(project_id=80e5b6ce14ab47dfac99cc80153d7735)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:56.969411 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1bb8a3a-39f4-499d-ad9b-4d518b6936bb admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:56.970826 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=92,keystone:DELETE=5,keystone:INSERT=3 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:37:57.023356 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-f1bb8a3a-39f4-499d-ad9b-4d518b6936bb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '80e5b6ce14ab47dfac99cc80153d7735', 'request_id': 'req-f1bb8a3a-39f4-499d-ad9b-4d518b6936bb'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:57.031068 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-f1bb8a3a-39f4-499d-ad9b-4d518b6936bb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '80e5b6ce14ab47dfac99cc80153d7735', 'request_id': 'req-f1bb8a3a-39f4-499d-ad9b-4d518b6936bb'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:57.036381 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-f1bb8a3a-39f4-499d-ad9b-4d518b6936bb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '80e5b6ce14ab47dfac99cc80153d7735', 'request_id': 'req-f1bb8a3a-39f4-499d-ad9b-4d518b6936bb'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:57.047757 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-f1bb8a3a-39f4-499d-ad9b-4d518b6936bb admin admin] The token cache is being invalidate because project 80e5b6ce14ab47dfac99cc80153d7735 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81434) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:37:57.048041 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-f1bb8a3a-39f4-499d-ad9b-4d518b6936bb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:57.048695 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-f1bb8a3a-39f4-499d-ad9b-4d518b6936bb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:37:57.050173 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1079/2134] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:37:56 2022] DELETE /identity/v3/projects/80e5b6ce14ab47dfac99cc80153d7735 => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:37:57.055249 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61067af2-b74c-4cd7-8b73-4a53adaa67bd None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:57.055542 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61067af2-b74c-4cd7-8b73-4a53adaa67bd None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:57.055821 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61067af2-b74c-4cd7-8b73-4a53adaa67bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:57.076334 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-61067af2-b74c-4cd7-8b73-4a53adaa67bd None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:37:57.077595 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-61067af2-b74c-4cd7-8b73-4a53adaa67bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:37:57.140344 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1056/2135] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:37:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:37:57.148028 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3451a98-914e-4156-b923-3afb35420f47 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:57.148316 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3451a98-914e-4156-b923-3afb35420f47 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:57.148592 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3451a98-914e-4156-b923-3afb35420f47 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:57.168878 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-a3451a98-914e-4156-b923-3afb35420f47 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:37:57.170119 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-a3451a98-914e-4156-b923-3afb35420f47 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:37:57.178819 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1080/2136] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:37:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:37:57.182970 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61067af2-b74c-4cd7-8b73-4a53adaa67bd None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:57.194468 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61067af2-b74c-4cd7-8b73-4a53adaa67bd None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:57.199975 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-723af3ef-75cd-41a0-82e1-0070472d8cbb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:57.200603 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-723af3ef-75cd-41a0-82e1-0070472d8cbb admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:57.200919 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-723af3ef-75cd-41a0-82e1-0070472d8cbb admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:57.201198 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-723af3ef-75cd-41a0-82e1-0070472d8cbb admin admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:57.201848 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-723af3ef-75cd-41a0-82e1-0070472d8cbb admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:37:57.202266 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-723af3ef-75cd-41a0-82e1-0070472d8cbb admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:57.203890 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-723af3ef-75cd-41a0-82e1-0070472d8cbb admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:57.217672 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1057/2137] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Sep 25 04:37:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:37:57.223082 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3451a98-914e-4156-b923-3afb35420f47 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:57.241485 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3451a98-914e-4156-b923-3afb35420f47 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:57.246548 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59ac838f-4139-42cc-a048-9d13d470387a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:57.247153 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59ac838f-4139-42cc-a048-9d13d470387a admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:57.247427 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59ac838f-4139-42cc-a048-9d13d470387a admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:57.247712 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59ac838f-4139-42cc-a048-9d13d470387a admin admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:57.248323 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59ac838f-4139-42cc-a048-9d13d470387a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:57.250273 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59ac838f-4139-42cc-a048-9d13d470387a admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:57.278761 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1081/2138] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:37:57 2022] POST /identity/v3/projects => generated 367 bytes in 56 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:37:57.282887 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-723af3ef-75cd-41a0-82e1-0070472d8cbb admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:57.293043 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-723af3ef-75cd-41a0-82e1-0070472d8cbb admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:57.348280 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2027ef3-6398-4378-8ccd-e914453f0abf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:57.348905 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2027ef3-6398-4378-8ccd-e914453f0abf admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:57.349188 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2027ef3-6398-4378-8ccd-e914453f0abf admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:57.349463 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2027ef3-6398-4378-8ccd-e914453f0abf admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:57.350100 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2027ef3-6398-4378-8ccd-e914453f0abf admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:57.352009 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2027ef3-6398-4378-8ccd-e914453f0abf admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:57.374609 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1058/2139] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:37:57 2022] POST /identity/v3/users => generated 284 bytes in 91 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 04:37:57.377189 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59ac838f-4139-42cc-a048-9d13d470387a admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:57.387915 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59ac838f-4139-42cc-a048-9d13d470387a admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:57.392611 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3efd904e-be63-4417-9c12-6975d2af3960 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:57.393248 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3efd904e-be63-4417-9c12-6975d2af3960 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:57.393531 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3efd904e-be63-4417-9c12-6975d2af3960 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:57.393812 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3efd904e-be63-4417-9c12-6975d2af3960 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:57.394454 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3efd904e-be63-4417-9c12-6975d2af3960 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:57.396040 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3efd904e-be63-4417-9c12-6975d2af3960 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:57.405930 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1082/2140] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:37:57 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:37:57.409770 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2027ef3-6398-4378-8ccd-e914453f0abf admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:57.424714 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2027ef3-6398-4378-8ccd-e914453f0abf admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:57.429617 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32fb0610-f1a4-456c-be81-1965d824c5db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:57.430250 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32fb0610-f1a4-456c-be81-1965d824c5db admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:57.430526 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32fb0610-f1a4-456c-be81-1965d824c5db admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:57.430796 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32fb0610-f1a4-456c-be81-1965d824c5db admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:57.431489 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32fb0610-f1a4-456c-be81-1965d824c5db admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:57.434174 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32fb0610-f1a4-456c-be81-1965d824c5db admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:57.440050 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:DELETE=9,keystone:SELECT=87,keystone:INSERT=4 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:37:57.446789 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1059/2141] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:37:57 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:37:57.451875 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3efd904e-be63-4417-9c12-6975d2af3960 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:57.462402 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3efd904e-be63-4417-9c12-6975d2af3960 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:57.467414 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea77f23e-a787-4f5c-b392-b663480503d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:57.468072 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea77f23e-a787-4f5c-b392-b663480503d8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:57.468357 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea77f23e-a787-4f5c-b392-b663480503d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:57.468668 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea77f23e-a787-4f5c-b392-b663480503d8 admin admin] PATH_INFO: `/v3/projects/9eb3b2316d404443b35b0d9509b7ca8a/users/893c10075da14df3a0d6d6c8ee07e56b/roles/8f6cc574056c4278bc9778f14dc4b411` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:57.484729 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea77f23e-a787-4f5c-b392-b663480503d8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9eb3b2316d404443b35b0d9509b7ca8a, user_id=893c10075da14df3a0d6d6c8ee07e56b, role_id=8f6cc574056c4278bc9778f14dc4b411)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:57.487440 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea77f23e-a787-4f5c-b392-b663480503d8 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:57.501221 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1083/2142] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:37:57 2022] PUT /identity/v3/projects/9eb3b2316d404443b35b0d9509b7ca8a/users/893c10075da14df3a0d6d6c8ee07e56b/roles/8f6cc574056c4278bc9778f14dc4b411 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:37:57.513397 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1268b1f3-7d30-4fef-959f-e2aa8b2a1391 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:57.513770 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1268b1f3-7d30-4fef-959f-e2aa8b2a1391 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:57.514090 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1268b1f3-7d30-4fef-959f-e2aa8b2a1391 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:57.542156 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-1268b1f3-7d30-4fef-959f-e2aa8b2a1391 None None] MFA Rules not processed for user `893c10075da14df3a0d6d6c8ee07e56b`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:37:57.543524 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-1268b1f3-7d30-4fef-959f-e2aa8b2a1391 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:37:57.608417 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1060/2143] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:37:57 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:37:57.619109 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea77f23e-a787-4f5c-b392-b663480503d8 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:37:57.621775 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-ea77f23e-a787-4f5c-b392-b663480503d8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:37:57.697892 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea77f23e-a787-4f5c-b392-b663480503d8 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:37:57.701903 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dedf340d-101c-4e1a-8af3-0fa84619425a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ffbc136c77b84e8c87128f7247f4ec03', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:37:57.702516 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dedf340d-101c-4e1a-8af3-0fa84619425a service cinder] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:37:57.702800 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dedf340d-101c-4e1a-8af3-0fa84619425a service cinder] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:37:57.703071 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dedf340d-101c-4e1a-8af3-0fa84619425a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:37:57.712050 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dedf340d-101c-4e1a-8af3-0fa84619425a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:37:57.713569 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dedf340d-101c-4e1a-8af3-0fa84619425a service cinder] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:37:57.723555 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1084/2144] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:37:57 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:38:00.125432 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2137f8b-54c4-4d62-ba00-861b9b9895b3 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:00.125878 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2137f8b-54c4-4d62-ba00-861b9b9895b3 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:00.125878 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2137f8b-54c4-4d62-ba00-861b9b9895b3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:00.146109 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-c2137f8b-54c4-4d62-ba00-861b9b9895b3 None None] MFA Rules not processed for user `329e3f9810b648218dd45a8b489adb1d`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:38:00.147334 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-c2137f8b-54c4-4d62-ba00-861b9b9895b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:38:00.226480 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1061/2145] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:38:00 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:38:07.724296 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=77,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:38:08.437683 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dedf340d-101c-4e1a-8af3-0fa84619425a service cinder] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:08.511943 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dedf340d-101c-4e1a-8af3-0fa84619425a service cinder] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:08.516571 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3448c8e-3732-4466-b043-ecbc1ba17ad6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:08.517295 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3448c8e-3732-4466-b043-ecbc1ba17ad6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:08.517599 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3448c8e-3732-4466-b043-ecbc1ba17ad6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:08.517924 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3448c8e-3732-4466-b043-ecbc1ba17ad6 admin admin] PATH_INFO: `/v3/users/893c10075da14df3a0d6d6c8ee07e56b` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:08.519238 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3448c8e-3732-4466-b043-ecbc1ba17ad6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=893c10075da14df3a0d6d6c8ee07e56b)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:08.521286 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3448c8e-3732-4466-b043-ecbc1ba17ad6 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:08.569648 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-c3448c8e-3732-4466-b043-ecbc1ba17ad6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '893c10075da14df3a0d6d6c8ee07e56b'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:08.577029 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-c3448c8e-3732-4466-b043-ecbc1ba17ad6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '893c10075da14df3a0d6d6c8ee07e56b'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:08.591434 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-c3448c8e-3732-4466-b043-ecbc1ba17ad6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '893c10075da14df3a0d6d6c8ee07e56b'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:08.592142 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-c3448c8e-3732-4466-b043-ecbc1ba17ad6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '893c10075da14df3a0d6d6c8ee07e56b'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:08.599828 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-c3448c8e-3732-4466-b043-ecbc1ba17ad6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '893c10075da14df3a0d6d6c8ee07e56b'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:08.606539 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1085/2146] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:38:08 2022] DELETE /identity/v3/users/893c10075da14df3a0d6d6c8ee07e56b => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:38:08.610828 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-018fbae8-07cf-49b6-8d58-565a0071bc89 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:08.611298 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-018fbae8-07cf-49b6-8d58-565a0071bc89 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:08.611515 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-018fbae8-07cf-49b6-8d58-565a0071bc89 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:08.631873 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-018fbae8-07cf-49b6-8d58-565a0071bc89 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:38:08.633113 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-018fbae8-07cf-49b6-8d58-565a0071bc89 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:38:08.704155 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1062/2147] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:38:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:38:08.714784 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3448c8e-3732-4466-b043-ecbc1ba17ad6 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:08.716241 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-c3448c8e-3732-4466-b043-ecbc1ba17ad6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:38:08.773206 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3448c8e-3732-4466-b043-ecbc1ba17ad6 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:08.778412 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d808c93e-e5fd-436b-bf27-02af54ffc8c5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:08.779080 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d808c93e-e5fd-436b-bf27-02af54ffc8c5 service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:08.779471 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d808c93e-e5fd-436b-bf27-02af54ffc8c5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:08.779783 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d808c93e-e5fd-436b-bf27-02af54ffc8c5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:08.789166 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d808c93e-e5fd-436b-bf27-02af54ffc8c5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:08.790638 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d808c93e-e5fd-436b-bf27-02af54ffc8c5 service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:08.800738 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1086/2148] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:38:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 25 04:38:09.129810 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-018fbae8-07cf-49b6-8d58-565a0071bc89 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:09.131606 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-018fbae8-07cf-49b6-8d58-565a0071bc89 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:38:09.144427 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-018fbae8-07cf-49b6-8d58-565a0071bc89 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:09.149640 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d79766dd-1b48-4dab-8c83-74e1f14ff3b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:09.150276 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d79766dd-1b48-4dab-8c83-74e1f14ff3b8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:09.150583 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d79766dd-1b48-4dab-8c83-74e1f14ff3b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:09.150858 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d79766dd-1b48-4dab-8c83-74e1f14ff3b8 admin admin] PATH_INFO: `/v3/projects/9eb3b2316d404443b35b0d9509b7ca8a` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:09.152609 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d79766dd-1b48-4dab-8c83-74e1f14ff3b8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9eb3b2316d404443b35b0d9509b7ca8a)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:09.154641 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d79766dd-1b48-4dab-8c83-74e1f14ff3b8 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:09.189896 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=98,keystone:DELETE=2 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:38:09.203946 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-d79766dd-1b48-4dab-8c83-74e1f14ff3b8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9eb3b2316d404443b35b0d9509b7ca8a', 'request_id': 'req-d79766dd-1b48-4dab-8c83-74e1f14ff3b8'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:09.211005 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-d79766dd-1b48-4dab-8c83-74e1f14ff3b8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9eb3b2316d404443b35b0d9509b7ca8a', 'request_id': 'req-d79766dd-1b48-4dab-8c83-74e1f14ff3b8'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:09.217589 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-d79766dd-1b48-4dab-8c83-74e1f14ff3b8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9eb3b2316d404443b35b0d9509b7ca8a', 'request_id': 'req-d79766dd-1b48-4dab-8c83-74e1f14ff3b8'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:09.228760 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-d79766dd-1b48-4dab-8c83-74e1f14ff3b8 admin admin] The token cache is being invalidate because project 9eb3b2316d404443b35b0d9509b7ca8a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:38:09.229114 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-d79766dd-1b48-4dab-8c83-74e1f14ff3b8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:09.229892 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-d79766dd-1b48-4dab-8c83-74e1f14ff3b8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:09.231384 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1063/2149] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:38:09 2022] DELETE /identity/v3/projects/9eb3b2316d404443b35b0d9509b7ca8a => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:38:09.236314 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b7cf347-ee1a-4e86-8c71-15713d2f238b None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:09.236609 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b7cf347-ee1a-4e86-8c71-15713d2f238b None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:09.236922 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b7cf347-ee1a-4e86-8c71-15713d2f238b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:09.256275 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-9b7cf347-ee1a-4e86-8c71-15713d2f238b None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:38:09.257525 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-9b7cf347-ee1a-4e86-8c71-15713d2f238b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:38:09.275429 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:38:09.320125 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1087/2150] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:38:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:38:09.329297 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9695790-ed32-49bc-8431-6ba25d9b093e None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:09.329604 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9695790-ed32-49bc-8431-6ba25d9b093e None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:09.329884 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9695790-ed32-49bc-8431-6ba25d9b093e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:09.351526 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-e9695790-ed32-49bc-8431-6ba25d9b093e None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:38:09.352759 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-e9695790-ed32-49bc-8431-6ba25d9b093e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:38:09.360922 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1064/2151] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:38:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:38:09.365110 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b7cf347-ee1a-4e86-8c71-15713d2f238b None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:09.375149 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b7cf347-ee1a-4e86-8c71-15713d2f238b None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:09.379723 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90153d12-e952-4c6e-9419-fce763e438db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:09.380322 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90153d12-e952-4c6e-9419-fce763e438db admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:09.380597 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90153d12-e952-4c6e-9419-fce763e438db admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:09.380893 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90153d12-e952-4c6e-9419-fce763e438db admin admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:09.381534 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90153d12-e952-4c6e-9419-fce763e438db admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:38:09.381832 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90153d12-e952-4c6e-9419-fce763e438db admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:09.383408 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90153d12-e952-4c6e-9419-fce763e438db admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:09.395353 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1088/2152] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Sep 25 04:38:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:38:09.399624 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9695790-ed32-49bc-8431-6ba25d9b093e None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:09.410171 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9695790-ed32-49bc-8431-6ba25d9b093e None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:09.414891 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03b02111-4364-4f78-9924-681cbf99b4bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:09.415485 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03b02111-4364-4f78-9924-681cbf99b4bc admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:09.415764 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03b02111-4364-4f78-9924-681cbf99b4bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:09.416034 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03b02111-4364-4f78-9924-681cbf99b4bc admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:09.416735 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03b02111-4364-4f78-9924-681cbf99b4bc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:09.418629 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03b02111-4364-4f78-9924-681cbf99b4bc admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:09.446972 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1065/2153] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:38:09 2022] POST /identity/v3/projects => generated 379 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:38:09.452629 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90153d12-e952-4c6e-9419-fce763e438db admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:09.464192 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90153d12-e952-4c6e-9419-fce763e438db admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:09.556901 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8db73e61-3f48-41cd-ad53-d35045ecf9ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:09.558229 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8db73e61-3f48-41cd-ad53-d35045ecf9ee admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:09.558553 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8db73e61-3f48-41cd-ad53-d35045ecf9ee admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:09.558839 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8db73e61-3f48-41cd-ad53-d35045ecf9ee admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:09.559477 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8db73e61-3f48-41cd-ad53-d35045ecf9ee admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:09.564153 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8db73e61-3f48-41cd-ad53-d35045ecf9ee admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:09.587896 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1089/2154] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:38:09 2022] POST /identity/v3/users => generated 290 bytes in 136 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:38:09.592600 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03b02111-4364-4f78-9924-681cbf99b4bc admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:09.603658 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03b02111-4364-4f78-9924-681cbf99b4bc admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:09.608878 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-575ad7b6-0e6d-4136-85f1-1fd036624509 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:09.609490 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-575ad7b6-0e6d-4136-85f1-1fd036624509 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:09.609779 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-575ad7b6-0e6d-4136-85f1-1fd036624509 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:09.610054 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-575ad7b6-0e6d-4136-85f1-1fd036624509 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:09.610697 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-575ad7b6-0e6d-4136-85f1-1fd036624509 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:09.612303 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-575ad7b6-0e6d-4136-85f1-1fd036624509 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:09.623998 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1066/2155] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:38:09 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:38:09.627839 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8db73e61-3f48-41cd-ad53-d35045ecf9ee admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:09.639045 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8db73e61-3f48-41cd-ad53-d35045ecf9ee admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:09.644016 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fba83055-025f-4be4-8d8b-6d8bcaa11343 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:09.644745 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba83055-025f-4be4-8d8b-6d8bcaa11343 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:09.645027 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba83055-025f-4be4-8d8b-6d8bcaa11343 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:09.645339 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba83055-025f-4be4-8d8b-6d8bcaa11343 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:09.646018 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fba83055-025f-4be4-8d8b-6d8bcaa11343 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:09.647723 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fba83055-025f-4be4-8d8b-6d8bcaa11343 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:09.658888 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1090/2156] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:38:09 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:38:09.663066 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-575ad7b6-0e6d-4136-85f1-1fd036624509 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:09.673508 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-575ad7b6-0e6d-4136-85f1-1fd036624509 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:09.678777 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c20b98b2-0d35-46dd-a750-e1e5ee35dc1e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:09.679902 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c20b98b2-0d35-46dd-a750-e1e5ee35dc1e admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:09.679902 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c20b98b2-0d35-46dd-a750-e1e5ee35dc1e admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:09.680105 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c20b98b2-0d35-46dd-a750-e1e5ee35dc1e admin admin] PATH_INFO: `/v3/projects/8d9ade84ba3340359470f619ac88696f/users/973fac2c848f467a910957638031e0ad/roles/8f6cc574056c4278bc9778f14dc4b411` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:09.696226 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c20b98b2-0d35-46dd-a750-e1e5ee35dc1e admin admin] RBAC: Authorizing `identity:create_grant(project_id=8d9ade84ba3340359470f619ac88696f, user_id=973fac2c848f467a910957638031e0ad, role_id=8f6cc574056c4278bc9778f14dc4b411)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:09.700021 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c20b98b2-0d35-46dd-a750-e1e5ee35dc1e admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:09.716822 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1067/2157] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:38:09 2022] PUT /identity/v3/projects/8d9ade84ba3340359470f619ac88696f/users/973fac2c848f467a910957638031e0ad/roles/8f6cc574056c4278bc9778f14dc4b411 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:38:09.727160 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b70f7aa5-21ff-4372-b9ee-a684f1884af0 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:09.727451 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b70f7aa5-21ff-4372-b9ee-a684f1884af0 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:09.727722 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b70f7aa5-21ff-4372-b9ee-a684f1884af0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:09.761486 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-b70f7aa5-21ff-4372-b9ee-a684f1884af0 None None] MFA Rules not processed for user `973fac2c848f467a910957638031e0ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:38:09.762727 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-b70f7aa5-21ff-4372-b9ee-a684f1884af0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:38:09.831975 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1091/2158] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:38:09 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:38:09.836862 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c20b98b2-0d35-46dd-a750-e1e5ee35dc1e admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:09.917536 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c20b98b2-0d35-46dd-a750-e1e5ee35dc1e admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:09.923060 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6366eb0-3587-4f21-a7a1-c3e0e473d665 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:09.923952 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6366eb0-3587-4f21-a7a1-c3e0e473d665 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:09.924076 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6366eb0-3587-4f21-a7a1-c3e0e473d665 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:09.924474 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6366eb0-3587-4f21-a7a1-c3e0e473d665 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:09.925195 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6366eb0-3587-4f21-a7a1-c3e0e473d665 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:09.927551 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6366eb0-3587-4f21-a7a1-c3e0e473d665 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:09.984738 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1068/2159] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:38:09 2022] POST /identity/v3/projects => generated 379 bytes in 139 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:38:09.984839 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b70f7aa5-21ff-4372-b9ee-a684f1884af0 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:09.992766 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b70f7aa5-21ff-4372-b9ee-a684f1884af0 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:10.009199 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:38:10.077134 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-443cf514-e6b6-4ac0-a62e-5aa7b1882406 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:10.077918 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-443cf514-e6b6-4ac0-a62e-5aa7b1882406 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:10.078197 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-443cf514-e6b6-4ac0-a62e-5aa7b1882406 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:10.078470 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-443cf514-e6b6-4ac0-a62e-5aa7b1882406 admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:10.079443 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-443cf514-e6b6-4ac0-a62e-5aa7b1882406 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:10.081126 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-443cf514-e6b6-4ac0-a62e-5aa7b1882406 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:10.106276 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1092/2160] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:38:09 2022] POST /identity/v3/users => generated 290 bytes in 127 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:38:10.110748 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6366eb0-3587-4f21-a7a1-c3e0e473d665 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:10.123844 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6366eb0-3587-4f21-a7a1-c3e0e473d665 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:10.146175 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aba3ada-6498-4051-acf8-6eaaf585b318 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:10.146822 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aba3ada-6498-4051-acf8-6eaaf585b318 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:10.147104 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aba3ada-6498-4051-acf8-6eaaf585b318 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:10.147400 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aba3ada-6498-4051-acf8-6eaaf585b318 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:10.148054 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aba3ada-6498-4051-acf8-6eaaf585b318 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:10.150526 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aba3ada-6498-4051-acf8-6eaaf585b318 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:10.164843 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1069/2161] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:38:10 2022] GET /identity/v3/roles => generated 1420 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:38:10.170257 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-443cf514-e6b6-4ac0-a62e-5aa7b1882406 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:10.181138 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-443cf514-e6b6-4ac0-a62e-5aa7b1882406 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:10.188923 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce7dad7c-eb35-4883-abfc-031235dc7411 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:10.189407 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce7dad7c-eb35-4883-abfc-031235dc7411 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:10.190371 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce7dad7c-eb35-4883-abfc-031235dc7411 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:10.190371 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce7dad7c-eb35-4883-abfc-031235dc7411 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:10.191006 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce7dad7c-eb35-4883-abfc-031235dc7411 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:10.192785 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce7dad7c-eb35-4883-abfc-031235dc7411 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:10.204749 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1093/2162] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:38:10 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:38:10.209369 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aba3ada-6498-4051-acf8-6eaaf585b318 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:10.220612 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aba3ada-6498-4051-acf8-6eaaf585b318 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:10.225609 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f157342-50e1-43aa-a1dd-71746fe13280 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:10.226344 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f157342-50e1-43aa-a1dd-71746fe13280 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:10.226697 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f157342-50e1-43aa-a1dd-71746fe13280 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:10.227043 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f157342-50e1-43aa-a1dd-71746fe13280 admin admin] PATH_INFO: `/v3/projects/b32bd4ca5df741b293c19c96b91714b8/users/eca07aac141d435c9796ea2d6e83eb01/roles/8f6cc574056c4278bc9778f14dc4b411` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:10.242522 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f157342-50e1-43aa-a1dd-71746fe13280 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b32bd4ca5df741b293c19c96b91714b8, user_id=eca07aac141d435c9796ea2d6e83eb01, role_id=8f6cc574056c4278bc9778f14dc4b411)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:10.245905 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f157342-50e1-43aa-a1dd-71746fe13280 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:10.262159 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1070/2163] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:38:10 2022] PUT /identity/v3/projects/b32bd4ca5df741b293c19c96b91714b8/users/eca07aac141d435c9796ea2d6e83eb01/roles/8f6cc574056c4278bc9778f14dc4b411 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:38:10.274829 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2cfacc6-cecd-4a01-af7f-04d0fa5e6a97 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:10.275125 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2cfacc6-cecd-4a01-af7f-04d0fa5e6a97 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:10.275413 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2cfacc6-cecd-4a01-af7f-04d0fa5e6a97 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:10.307006 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-e2cfacc6-cecd-4a01-af7f-04d0fa5e6a97 None None] MFA Rules not processed for user `eca07aac141d435c9796ea2d6e83eb01`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:38:10.309227 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-e2cfacc6-cecd-4a01-af7f-04d0fa5e6a97 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:38:10.375543 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1094/2164] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:38:10 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:38:10.380242 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f157342-50e1-43aa-a1dd-71746fe13280 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:10.389069 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=90,keystone:DELETE=5,keystone:INSERT=5 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:38:10.450849 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f157342-50e1-43aa-a1dd-71746fe13280 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:10.456517 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-164526d7-7e06-456b-a37b-9930b1998aba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:10.457248 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-164526d7-7e06-456b-a37b-9930b1998aba admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:10.457532 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-164526d7-7e06-456b-a37b-9930b1998aba admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:10.457807 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-164526d7-7e06-456b-a37b-9930b1998aba admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:10.458834 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-164526d7-7e06-456b-a37b-9930b1998aba admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:10.461143 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-164526d7-7e06-456b-a37b-9930b1998aba admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:10.495759 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1071/2165] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:38:10 2022] POST /identity/v3/projects => generated 379 bytes in 116 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 04:38:10.499900 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2cfacc6-cecd-4a01-af7f-04d0fa5e6a97 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:10.511309 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2cfacc6-cecd-4a01-af7f-04d0fa5e6a97 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:10.570519 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d7ad659-e1f8-4ece-ba78-b4fcdb345b8b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:10.571212 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d7ad659-e1f8-4ece-ba78-b4fcdb345b8b admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:10.571488 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d7ad659-e1f8-4ece-ba78-b4fcdb345b8b admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:10.571832 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d7ad659-e1f8-4ece-ba78-b4fcdb345b8b admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:10.572506 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d7ad659-e1f8-4ece-ba78-b4fcdb345b8b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:10.574645 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d7ad659-e1f8-4ece-ba78-b4fcdb345b8b admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:10.596612 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1095/2166] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:38:10 2022] POST /identity/v3/users => generated 296 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:38:10.600536 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-164526d7-7e06-456b-a37b-9930b1998aba admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:10.612113 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-164526d7-7e06-456b-a37b-9930b1998aba admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:10.616852 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fafae50-ff7e-44c4-8473-60badd74aff8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:10.617448 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fafae50-ff7e-44c4-8473-60badd74aff8 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:10.617746 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fafae50-ff7e-44c4-8473-60badd74aff8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:10.618001 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fafae50-ff7e-44c4-8473-60badd74aff8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:10.618689 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fafae50-ff7e-44c4-8473-60badd74aff8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:10.621115 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fafae50-ff7e-44c4-8473-60badd74aff8 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:10.633491 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1072/2167] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:38:10 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 25 04:38:10.638354 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d7ad659-e1f8-4ece-ba78-b4fcdb345b8b admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:10.648741 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d7ad659-e1f8-4ece-ba78-b4fcdb345b8b admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:10.653600 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e98d537d-da23-4e1a-ba92-386b4663f9de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:10.654272 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e98d537d-da23-4e1a-ba92-386b4663f9de admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:10.654594 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e98d537d-da23-4e1a-ba92-386b4663f9de admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:10.654872 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e98d537d-da23-4e1a-ba92-386b4663f9de admin admin] PATH_INFO: `/v3/projects/570d7a550a164b0dbea2092d62ed77b9/users/1ab61efad8b44f56bfc332f88c27521e/roles/c51fe7c5ad804040879acfb54269455f` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:10.668377 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:38:10.671739 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e98d537d-da23-4e1a-ba92-386b4663f9de admin admin] RBAC: Authorizing `identity:create_grant(project_id=570d7a550a164b0dbea2092d62ed77b9, user_id=1ab61efad8b44f56bfc332f88c27521e, role_id=c51fe7c5ad804040879acfb54269455f)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:10.675160 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e98d537d-da23-4e1a-ba92-386b4663f9de admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:10.689994 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1096/2168] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:38:10 2022] PUT /identity/v3/projects/570d7a550a164b0dbea2092d62ed77b9/users/1ab61efad8b44f56bfc332f88c27521e/roles/c51fe7c5ad804040879acfb54269455f => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:38:10.694324 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fafae50-ff7e-44c4-8473-60badd74aff8 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:10.704538 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fafae50-ff7e-44c4-8473-60badd74aff8 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:10.769873 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cc879da-090e-4072-ab97-b72c012ad1a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:10.770577 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cc879da-090e-4072-ab97-b72c012ad1a8 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:10.770974 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cc879da-090e-4072-ab97-b72c012ad1a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:10.771329 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cc879da-090e-4072-ab97-b72c012ad1a8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:10.772041 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cc879da-090e-4072-ab97-b72c012ad1a8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:10.773751 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cc879da-090e-4072-ab97-b72c012ad1a8 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:10.785252 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1073/2169] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:38:10 2022] GET /identity/v3/roles => generated 1420 bytes in 91 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:38:10.789415 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e98d537d-da23-4e1a-ba92-386b4663f9de admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:10.801076 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e98d537d-da23-4e1a-ba92-386b4663f9de admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:10.806247 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ea847ee-2e65-45c7-8ee3-709d9e008d10 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:10.806976 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ea847ee-2e65-45c7-8ee3-709d9e008d10 admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:10.807879 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ea847ee-2e65-45c7-8ee3-709d9e008d10 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:10.808420 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ea847ee-2e65-45c7-8ee3-709d9e008d10 admin admin] PATH_INFO: `/v3/projects/570d7a550a164b0dbea2092d62ed77b9/users/1ab61efad8b44f56bfc332f88c27521e/roles/c51fe7c5ad804040879acfb54269455f` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:10.810717 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ea847ee-2e65-45c7-8ee3-709d9e008d10 admin admin] RBAC: Authorizing `identity:create_grant(project_id=570d7a550a164b0dbea2092d62ed77b9, user_id=1ab61efad8b44f56bfc332f88c27521e, role_id=c51fe7c5ad804040879acfb54269455f)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:10.815831 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ea847ee-2e65-45c7-8ee3-709d9e008d10 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:10.826385 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1097/2170] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:38:10 2022] PUT /identity/v3/projects/570d7a550a164b0dbea2092d62ed77b9/users/1ab61efad8b44f56bfc332f88c27521e/roles/c51fe7c5ad804040879acfb54269455f => generated 0 bytes in 37 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:38:10.836954 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8773b3e-d55b-4424-96d0-120583b5bb93 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:10.837271 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8773b3e-d55b-4424-96d0-120583b5bb93 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:10.837553 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8773b3e-d55b-4424-96d0-120583b5bb93 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:10.867362 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-e8773b3e-d55b-4424-96d0-120583b5bb93 None None] MFA Rules not processed for user `1ab61efad8b44f56bfc332f88c27521e`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:38:10.868606 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-e8773b3e-d55b-4424-96d0-120583b5bb93 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:38:10.937833 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:38:10.946482 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1074/2171] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:38:10 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:38:10.954530 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ea847ee-2e65-45c7-8ee3-709d9e008d10 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:10.956814 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-3ea847ee-2e65-45c7-8ee3-709d9e008d10 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:38:11.018762 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ea847ee-2e65-45c7-8ee3-709d9e008d10 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:11.022533 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18aa3b95-5d2d-4e25-ad23-f46027ea8ea1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ffbc136c77b84e8c87128f7247f4ec03', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:11.023143 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18aa3b95-5d2d-4e25-ad23-f46027ea8ea1 service cinder] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:11.023425 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18aa3b95-5d2d-4e25-ad23-f46027ea8ea1 service cinder] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:11.023698 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18aa3b95-5d2d-4e25-ad23-f46027ea8ea1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:11.033858 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18aa3b95-5d2d-4e25-ad23-f46027ea8ea1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:11.035284 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18aa3b95-5d2d-4e25-ad23-f46027ea8ea1 service cinder] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:11.045746 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1098/2172] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:38:10 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:38:11.088103 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8773b3e-d55b-4424-96d0-120583b5bb93 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:11.090111 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-e8773b3e-d55b-4424-96d0-120583b5bb93 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:38:11.102560 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8773b3e-d55b-4424-96d0-120583b5bb93 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:11.107694 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52388730-d082-4ec2-bc32-6a650c2c8706 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ffbc136c77b84e8c87128f7247f4ec03', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:11.108291 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52388730-d082-4ec2-bc32-6a650c2c8706 service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:11.108672 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52388730-d082-4ec2-bc32-6a650c2c8706 service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:11.109090 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52388730-d082-4ec2-bc32-6a650c2c8706 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:11.172040 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52388730-d082-4ec2-bc32-6a650c2c8706 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:11.173503 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52388730-d082-4ec2-bc32-6a650c2c8706 service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:11.184093 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1075/2173] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:38:11 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:38:15.209355 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18aa3b95-5d2d-4e25-ad23-f46027ea8ea1 service cinder] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:15.290553 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18aa3b95-5d2d-4e25-ad23-f46027ea8ea1 service cinder] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:15.295339 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39125aed-d956-426a-b125-fce4ff2c21cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:15.296083 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39125aed-d956-426a-b125-fce4ff2c21cc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:15.296370 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39125aed-d956-426a-b125-fce4ff2c21cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:15.296672 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39125aed-d956-426a-b125-fce4ff2c21cc admin admin] PATH_INFO: `/v3/users/973fac2c848f467a910957638031e0ad` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:15.297820 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39125aed-d956-426a-b125-fce4ff2c21cc admin admin] RBAC: Authorizing `identity:delete_user(user_id=973fac2c848f467a910957638031e0ad)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:15.299734 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39125aed-d956-426a-b125-fce4ff2c21cc admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:15.351817 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-39125aed-d956-426a-b125-fce4ff2c21cc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '973fac2c848f467a910957638031e0ad'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:15.361750 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-39125aed-d956-426a-b125-fce4ff2c21cc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '973fac2c848f467a910957638031e0ad'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:15.375629 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-39125aed-d956-426a-b125-fce4ff2c21cc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '973fac2c848f467a910957638031e0ad'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:15.376490 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-39125aed-d956-426a-b125-fce4ff2c21cc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '973fac2c848f467a910957638031e0ad'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:15.383826 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-39125aed-d956-426a-b125-fce4ff2c21cc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '973fac2c848f467a910957638031e0ad'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:15.386828 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=87,keystone:INSERT=3,keystone:DELETE=10 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:38:15.395618 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1099/2174] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:38:15 2022] DELETE /identity/v3/users/973fac2c848f467a910957638031e0ad => generated 0 bytes in 184 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:38:15.397865 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60c28a17-42c0-4bb3-be43-a42472cc6f5c None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:15.398816 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60c28a17-42c0-4bb3-be43-a42472cc6f5c None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:15.399294 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60c28a17-42c0-4bb3-be43-a42472cc6f5c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:15.421743 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-60c28a17-42c0-4bb3-be43-a42472cc6f5c None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:38:15.423073 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-60c28a17-42c0-4bb3-be43-a42472cc6f5c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:38:15.508711 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1076/2175] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:38:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:38:15.520013 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39125aed-d956-426a-b125-fce4ff2c21cc admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:15.521900 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-39125aed-d956-426a-b125-fce4ff2c21cc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:38:15.589140 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39125aed-d956-426a-b125-fce4ff2c21cc admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:15.593066 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c846009d-dfe3-4284-bac5-d827e9df5d73 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:15.593749 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c846009d-dfe3-4284-bac5-d827e9df5d73 service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:15.594029 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c846009d-dfe3-4284-bac5-d827e9df5d73 service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:15.594305 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c846009d-dfe3-4284-bac5-d827e9df5d73 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:15.604083 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c846009d-dfe3-4284-bac5-d827e9df5d73 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:15.605552 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c846009d-dfe3-4284-bac5-d827e9df5d73 service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:15.615257 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1100/2176] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:38:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:38:15.888646 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60c28a17-42c0-4bb3-be43-a42472cc6f5c None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:15.890329 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-60c28a17-42c0-4bb3-be43-a42472cc6f5c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:38:15.902700 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60c28a17-42c0-4bb3-be43-a42472cc6f5c None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:15.907786 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3db644d5-c844-41d6-a5bb-297b88fb0415 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:15.908517 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3db644d5-c844-41d6-a5bb-297b88fb0415 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:15.908900 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3db644d5-c844-41d6-a5bb-297b88fb0415 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:15.909250 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3db644d5-c844-41d6-a5bb-297b88fb0415 admin admin] PATH_INFO: `/v3/projects/8d9ade84ba3340359470f619ac88696f` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:15.910468 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3db644d5-c844-41d6-a5bb-297b88fb0415 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8d9ade84ba3340359470f619ac88696f)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:15.912570 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3db644d5-c844-41d6-a5bb-297b88fb0415 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:15.962387 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-3db644d5-c844-41d6-a5bb-297b88fb0415 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8d9ade84ba3340359470f619ac88696f', 'request_id': 'req-3db644d5-c844-41d6-a5bb-297b88fb0415'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:15.969755 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-3db644d5-c844-41d6-a5bb-297b88fb0415 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8d9ade84ba3340359470f619ac88696f', 'request_id': 'req-3db644d5-c844-41d6-a5bb-297b88fb0415'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:15.975806 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-3db644d5-c844-41d6-a5bb-297b88fb0415 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8d9ade84ba3340359470f619ac88696f', 'request_id': 'req-3db644d5-c844-41d6-a5bb-297b88fb0415'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:15.987851 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-3db644d5-c844-41d6-a5bb-297b88fb0415 admin admin] The token cache is being invalidate because project 8d9ade84ba3340359470f619ac88696f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:38:15.988145 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-3db644d5-c844-41d6-a5bb-297b88fb0415 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:15.988824 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-3db644d5-c844-41d6-a5bb-297b88fb0415 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:15.990321 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1077/2177] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:38:15 2022] DELETE /identity/v3/projects/8d9ade84ba3340359470f619ac88696f => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:38:15.994315 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c846009d-dfe3-4284-bac5-d827e9df5d73 service neutron] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:16.004849 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c846009d-dfe3-4284-bac5-d827e9df5d73 service neutron] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:16.061812 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38235e88-2727-4f4c-adda-0e7132219809 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:16.062458 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38235e88-2727-4f4c-adda-0e7132219809 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:16.062744 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38235e88-2727-4f4c-adda-0e7132219809 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:16.063017 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38235e88-2727-4f4c-adda-0e7132219809 admin admin] PATH_INFO: `/v3/users/eca07aac141d435c9796ea2d6e83eb01` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:16.064145 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38235e88-2727-4f4c-adda-0e7132219809 admin admin] RBAC: Authorizing `identity:delete_user(user_id=eca07aac141d435c9796ea2d6e83eb01)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:16.066105 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38235e88-2727-4f4c-adda-0e7132219809 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:16.111981 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-38235e88-2727-4f4c-adda-0e7132219809 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'eca07aac141d435c9796ea2d6e83eb01'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:16.119195 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-38235e88-2727-4f4c-adda-0e7132219809 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'eca07aac141d435c9796ea2d6e83eb01'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:16.131615 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-38235e88-2727-4f4c-adda-0e7132219809 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'eca07aac141d435c9796ea2d6e83eb01'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:16.132271 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-38235e88-2727-4f4c-adda-0e7132219809 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'eca07aac141d435c9796ea2d6e83eb01'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:16.139202 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-38235e88-2727-4f4c-adda-0e7132219809 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'eca07aac141d435c9796ea2d6e83eb01'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:16.145046 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1101/2178] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:38:15 2022] DELETE /identity/v3/users/eca07aac141d435c9796ea2d6e83eb01 => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:38:16.418484 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3db644d5-c844-41d6-a5bb-297b88fb0415 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:16.420003 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-3db644d5-c844-41d6-a5bb-297b88fb0415 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:38:16.434877 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3db644d5-c844-41d6-a5bb-297b88fb0415 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:16.455418 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:38:16.501279 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad8811de-e691-4917-ba48-0a23f69d1121 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:16.501931 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad8811de-e691-4917-ba48-0a23f69d1121 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:16.502234 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad8811de-e691-4917-ba48-0a23f69d1121 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:16.502508 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad8811de-e691-4917-ba48-0a23f69d1121 admin admin] PATH_INFO: `/v3/projects/b32bd4ca5df741b293c19c96b91714b8` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:16.503729 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad8811de-e691-4917-ba48-0a23f69d1121 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b32bd4ca5df741b293c19c96b91714b8)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:16.505673 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad8811de-e691-4917-ba48-0a23f69d1121 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:16.554015 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-ad8811de-e691-4917-ba48-0a23f69d1121 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b32bd4ca5df741b293c19c96b91714b8', 'request_id': 'req-ad8811de-e691-4917-ba48-0a23f69d1121'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:16.562489 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-ad8811de-e691-4917-ba48-0a23f69d1121 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b32bd4ca5df741b293c19c96b91714b8', 'request_id': 'req-ad8811de-e691-4917-ba48-0a23f69d1121'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:16.568243 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-ad8811de-e691-4917-ba48-0a23f69d1121 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b32bd4ca5df741b293c19c96b91714b8', 'request_id': 'req-ad8811de-e691-4917-ba48-0a23f69d1121'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:16.579659 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-ad8811de-e691-4917-ba48-0a23f69d1121 admin admin] The token cache is being invalidate because project b32bd4ca5df741b293c19c96b91714b8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:38:16.579950 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-ad8811de-e691-4917-ba48-0a23f69d1121 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:16.580574 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-ad8811de-e691-4917-ba48-0a23f69d1121 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:16.582399 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1078/2179] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:38:16 2022] DELETE /identity/v3/projects/b32bd4ca5df741b293c19c96b91714b8 => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:38:16.586804 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38235e88-2727-4f4c-adda-0e7132219809 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:16.589543 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-38235e88-2727-4f4c-adda-0e7132219809 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:38:16.602179 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38235e88-2727-4f4c-adda-0e7132219809 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:16.620760 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:38:16.661694 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f41ecc9e-5392-4e8c-9dc9-1ba47b3ac51f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:16.662332 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f41ecc9e-5392-4e8c-9dc9-1ba47b3ac51f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:16.662614 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f41ecc9e-5392-4e8c-9dc9-1ba47b3ac51f admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:16.662887 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f41ecc9e-5392-4e8c-9dc9-1ba47b3ac51f admin admin] PATH_INFO: `/v3/users/1ab61efad8b44f56bfc332f88c27521e` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:16.664027 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f41ecc9e-5392-4e8c-9dc9-1ba47b3ac51f admin admin] RBAC: Authorizing `identity:delete_user(user_id=1ab61efad8b44f56bfc332f88c27521e)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:16.665977 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f41ecc9e-5392-4e8c-9dc9-1ba47b3ac51f admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:16.714708 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-f41ecc9e-5392-4e8c-9dc9-1ba47b3ac51f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1ab61efad8b44f56bfc332f88c27521e'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:16.721938 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-f41ecc9e-5392-4e8c-9dc9-1ba47b3ac51f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1ab61efad8b44f56bfc332f88c27521e'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:16.735845 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-f41ecc9e-5392-4e8c-9dc9-1ba47b3ac51f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1ab61efad8b44f56bfc332f88c27521e'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:16.736571 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-f41ecc9e-5392-4e8c-9dc9-1ba47b3ac51f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1ab61efad8b44f56bfc332f88c27521e'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:16.743449 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-f41ecc9e-5392-4e8c-9dc9-1ba47b3ac51f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1ab61efad8b44f56bfc332f88c27521e'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:16.761264 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1102/2180] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:38:16 2022] DELETE /identity/v3/users/1ab61efad8b44f56bfc332f88c27521e => generated 0 bytes in 175 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:38:17.026146 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad8811de-e691-4917-ba48-0a23f69d1121 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:17.027813 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-ad8811de-e691-4917-ba48-0a23f69d1121 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:38:17.043849 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad8811de-e691-4917-ba48-0a23f69d1121 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:17.103154 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32128591-5f3b-4742-b003-25731cab48ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:17.103715 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32128591-5f3b-4742-b003-25731cab48ff admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:17.104006 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32128591-5f3b-4742-b003-25731cab48ff admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:17.104277 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32128591-5f3b-4742-b003-25731cab48ff admin admin] PATH_INFO: `/v3/projects/570d7a550a164b0dbea2092d62ed77b9` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:17.105445 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32128591-5f3b-4742-b003-25731cab48ff admin admin] RBAC: Authorizing `identity:delete_project(project_id=570d7a550a164b0dbea2092d62ed77b9)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:17.107368 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32128591-5f3b-4742-b003-25731cab48ff admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:17.153606 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-32128591-5f3b-4742-b003-25731cab48ff admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '570d7a550a164b0dbea2092d62ed77b9', 'request_id': 'req-32128591-5f3b-4742-b003-25731cab48ff'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:17.157113 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:38:17.161518 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-32128591-5f3b-4742-b003-25731cab48ff admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '570d7a550a164b0dbea2092d62ed77b9', 'request_id': 'req-32128591-5f3b-4742-b003-25731cab48ff'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:17.168735 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-32128591-5f3b-4742-b003-25731cab48ff admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '570d7a550a164b0dbea2092d62ed77b9', 'request_id': 'req-32128591-5f3b-4742-b003-25731cab48ff'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:17.179796 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-32128591-5f3b-4742-b003-25731cab48ff admin admin] The token cache is being invalidate because project 570d7a550a164b0dbea2092d62ed77b9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:38:17.180090 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-32128591-5f3b-4742-b003-25731cab48ff admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:17.180802 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-32128591-5f3b-4742-b003-25731cab48ff admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:17.182940 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1079/2181] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:38:17 2022] DELETE /identity/v3/projects/570d7a550a164b0dbea2092d62ed77b9 => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:38:17.189816 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-960e44a2-4eaa-4fdb-9071-dc3a49e3b391 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:17.190325 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-960e44a2-4eaa-4fdb-9071-dc3a49e3b391 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:17.190689 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-960e44a2-4eaa-4fdb-9071-dc3a49e3b391 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:17.211303 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-960e44a2-4eaa-4fdb-9071-dc3a49e3b391 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:38:17.212500 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-960e44a2-4eaa-4fdb-9071-dc3a49e3b391 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:38:17.270539 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1103/2182] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:38:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:38:17.279074 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-986fa179-189c-493d-b214-1b481d2593fc None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:17.279368 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-986fa179-189c-493d-b214-1b481d2593fc None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:17.279647 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-986fa179-189c-493d-b214-1b481d2593fc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:17.299576 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-986fa179-189c-493d-b214-1b481d2593fc None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:38:17.300846 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-986fa179-189c-493d-b214-1b481d2593fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:38:17.309443 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1080/2183] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:38:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:38:17.313544 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-960e44a2-4eaa-4fdb-9071-dc3a49e3b391 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:17.323397 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-960e44a2-4eaa-4fdb-9071-dc3a49e3b391 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:17.328295 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1f29076-555a-44dc-a1f8-3f2a2e4420c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:17.328923 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1f29076-555a-44dc-a1f8-3f2a2e4420c5 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:17.329203 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1f29076-555a-44dc-a1f8-3f2a2e4420c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:17.329474 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1f29076-555a-44dc-a1f8-3f2a2e4420c5 admin admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:17.330112 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1f29076-555a-44dc-a1f8-3f2a2e4420c5 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:38:17.330415 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1f29076-555a-44dc-a1f8-3f2a2e4420c5 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:17.332004 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1f29076-555a-44dc-a1f8-3f2a2e4420c5 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:17.345448 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1104/2184] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Sep 25 04:38:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:38:17.349797 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-986fa179-189c-493d-b214-1b481d2593fc None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:17.359879 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-986fa179-189c-493d-b214-1b481d2593fc None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:17.364765 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd74fa87-58a7-472e-adbc-f7b16eb2d025 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:17.365465 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd74fa87-58a7-472e-adbc-f7b16eb2d025 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:17.365817 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd74fa87-58a7-472e-adbc-f7b16eb2d025 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:17.366168 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd74fa87-58a7-472e-adbc-f7b16eb2d025 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:17.366834 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd74fa87-58a7-472e-adbc-f7b16eb2d025 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:17.368769 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd74fa87-58a7-472e-adbc-f7b16eb2d025 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:17.399701 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1081/2185] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:38:17 2022] POST /identity/v3/projects => generated 385 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:38:17.402805 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1f29076-555a-44dc-a1f8-3f2a2e4420c5 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:17.412888 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1f29076-555a-44dc-a1f8-3f2a2e4420c5 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:17.439552 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:38:17.474887 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88c5844c-60a1-4bb5-8c67-a9cd02f71589 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:17.475793 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88c5844c-60a1-4bb5-8c67-a9cd02f71589 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:17.476076 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88c5844c-60a1-4bb5-8c67-a9cd02f71589 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:17.476349 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88c5844c-60a1-4bb5-8c67-a9cd02f71589 admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:17.477031 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88c5844c-60a1-4bb5-8c67-a9cd02f71589 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:17.479519 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88c5844c-60a1-4bb5-8c67-a9cd02f71589 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:17.501475 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1105/2186] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:38:17 2022] POST /identity/v3/users => generated 293 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 04:38:17.505595 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd74fa87-58a7-472e-adbc-f7b16eb2d025 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:17.516014 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd74fa87-58a7-472e-adbc-f7b16eb2d025 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:17.520559 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33fd741a-ce78-40de-89ca-674eed4b0c4e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:17.521179 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33fd741a-ce78-40de-89ca-674eed4b0c4e admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:17.521455 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33fd741a-ce78-40de-89ca-674eed4b0c4e admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:17.521723 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33fd741a-ce78-40de-89ca-674eed4b0c4e admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:17.522355 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33fd741a-ce78-40de-89ca-674eed4b0c4e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:17.523958 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33fd741a-ce78-40de-89ca-674eed4b0c4e admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:17.534013 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1082/2187] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:38:17 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:38:17.538343 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88c5844c-60a1-4bb5-8c67-a9cd02f71589 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:17.549117 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88c5844c-60a1-4bb5-8c67-a9cd02f71589 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:17.553842 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4de58703-8599-4c75-8d3c-b8dac65e6021 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:17.554448 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de58703-8599-4c75-8d3c-b8dac65e6021 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:17.554731 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de58703-8599-4c75-8d3c-b8dac65e6021 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:17.555007 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de58703-8599-4c75-8d3c-b8dac65e6021 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:17.555652 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4de58703-8599-4c75-8d3c-b8dac65e6021 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:17.557861 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4de58703-8599-4c75-8d3c-b8dac65e6021 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:17.568473 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1106/2188] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:38:17 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:38:17.572932 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33fd741a-ce78-40de-89ca-674eed4b0c4e admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:17.586147 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33fd741a-ce78-40de-89ca-674eed4b0c4e admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:17.591383 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d42e95f-9989-4442-98c6-02ac7735e6ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:17.592029 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d42e95f-9989-4442-98c6-02ac7735e6ec admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:17.592308 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d42e95f-9989-4442-98c6-02ac7735e6ec admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:17.592577 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d42e95f-9989-4442-98c6-02ac7735e6ec admin admin] PATH_INFO: `/v3/projects/0620da484fca4c4dab9423b3825abaa8/users/d9c92669a89e459488de8a6edda33363/roles/8f6cc574056c4278bc9778f14dc4b411` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:17.608574 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d42e95f-9989-4442-98c6-02ac7735e6ec admin admin] RBAC: Authorizing `identity:create_grant(project_id=0620da484fca4c4dab9423b3825abaa8, user_id=d9c92669a89e459488de8a6edda33363, role_id=8f6cc574056c4278bc9778f14dc4b411)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:17.611465 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d42e95f-9989-4442-98c6-02ac7735e6ec admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:17.625374 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1083/2189] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:38:17 2022] PUT /identity/v3/projects/0620da484fca4c4dab9423b3825abaa8/users/d9c92669a89e459488de8a6edda33363/roles/8f6cc574056c4278bc9778f14dc4b411 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 04:38:17.638705 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8848c8e3-8214-4dbd-bd17-c4face9f73ba None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:17.639321 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8848c8e3-8214-4dbd-bd17-c4face9f73ba None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:17.639409 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8848c8e3-8214-4dbd-bd17-c4face9f73ba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:17.669864 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-8848c8e3-8214-4dbd-bd17-c4face9f73ba None None] MFA Rules not processed for user `d9c92669a89e459488de8a6edda33363`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:38:17.671457 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-8848c8e3-8214-4dbd-bd17-c4face9f73ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:38:17.741956 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1107/2190] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:38:17 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:38:17.746623 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d42e95f-9989-4442-98c6-02ac7735e6ec admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:17.817244 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d42e95f-9989-4442-98c6-02ac7735e6ec admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:17.822837 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8cd118c-7f9b-44e4-9d98-3504efd8cf48 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:17.823479 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8cd118c-7f9b-44e4-9d98-3504efd8cf48 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:17.823765 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8cd118c-7f9b-44e4-9d98-3504efd8cf48 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:17.824041 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8cd118c-7f9b-44e4-9d98-3504efd8cf48 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:17.824727 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8cd118c-7f9b-44e4-9d98-3504efd8cf48 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:17.826639 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8cd118c-7f9b-44e4-9d98-3504efd8cf48 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:17.855524 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1084/2191] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:38:17 2022] POST /identity/v3/projects => generated 385 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:38:17.861373 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8848c8e3-8214-4dbd-bd17-c4face9f73ba None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:17.873224 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8848c8e3-8214-4dbd-bd17-c4face9f73ba None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:17.939729 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b43b5628-8571-4a1e-9851-5f517d7063e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:17.940427 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b43b5628-8571-4a1e-9851-5f517d7063e2 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:17.940865 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b43b5628-8571-4a1e-9851-5f517d7063e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:17.941209 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b43b5628-8571-4a1e-9851-5f517d7063e2 admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:17.941916 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b43b5628-8571-4a1e-9851-5f517d7063e2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:17.943925 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b43b5628-8571-4a1e-9851-5f517d7063e2 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:17.974221 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1108/2192] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:38:17 2022] POST /identity/v3/users => generated 299 bytes in 113 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:38:17.978539 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8cd118c-7f9b-44e4-9d98-3504efd8cf48 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:17.993814 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8cd118c-7f9b-44e4-9d98-3504efd8cf48 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:18.001117 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18b0f4bf-fdb2-432a-8ce1-4bcf09948325 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:18.001917 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18b0f4bf-fdb2-432a-8ce1-4bcf09948325 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:18.002288 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18b0f4bf-fdb2-432a-8ce1-4bcf09948325 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:18.003274 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18b0f4bf-fdb2-432a-8ce1-4bcf09948325 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:18.004036 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18b0f4bf-fdb2-432a-8ce1-4bcf09948325 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:18.007096 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18b0f4bf-fdb2-432a-8ce1-4bcf09948325 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:18.021393 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1085/2193] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:38:17 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:38:18.025256 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b43b5628-8571-4a1e-9851-5f517d7063e2 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:18.037188 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b43b5628-8571-4a1e-9851-5f517d7063e2 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:18.049100 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd7ff625-b37e-4d9c-9ce8-c0d5adc61862 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:18.049859 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd7ff625-b37e-4d9c-9ce8-c0d5adc61862 admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:18.050245 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd7ff625-b37e-4d9c-9ce8-c0d5adc61862 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:18.050726 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd7ff625-b37e-4d9c-9ce8-c0d5adc61862 admin admin] PATH_INFO: `/v3/projects/9aaa942d093a4b70976fdd342e16a17b/users/ddbcee24fcdc46c684343de16898ff62/roles/c51fe7c5ad804040879acfb54269455f` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:18.071129 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd7ff625-b37e-4d9c-9ce8-c0d5adc61862 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9aaa942d093a4b70976fdd342e16a17b, user_id=ddbcee24fcdc46c684343de16898ff62, role_id=c51fe7c5ad804040879acfb54269455f)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:18.074700 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd7ff625-b37e-4d9c-9ce8-c0d5adc61862 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:18.090305 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1109/2194] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:38:18 2022] PUT /identity/v3/projects/9aaa942d093a4b70976fdd342e16a17b/users/ddbcee24fcdc46c684343de16898ff62/roles/c51fe7c5ad804040879acfb54269455f => generated 0 bytes in 66 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 04:38:18.095008 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18b0f4bf-fdb2-432a-8ce1-4bcf09948325 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:18.107083 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18b0f4bf-fdb2-432a-8ce1-4bcf09948325 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:18.157913 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=95,keystone:DELETE=2,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:38:18.169574 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87a57dcb-9f0b-4ba1-a6c1-5fafef284b4d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:18.170223 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a57dcb-9f0b-4ba1-a6c1-5fafef284b4d admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:18.170508 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a57dcb-9f0b-4ba1-a6c1-5fafef284b4d admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:18.170782 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a57dcb-9f0b-4ba1-a6c1-5fafef284b4d admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:18.171430 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87a57dcb-9f0b-4ba1-a6c1-5fafef284b4d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:18.173078 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87a57dcb-9f0b-4ba1-a6c1-5fafef284b4d admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:18.184681 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1086/2195] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:38:18 2022] GET /identity/v3/roles => generated 1420 bytes in 91 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:38:18.188835 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd7ff625-b37e-4d9c-9ce8-c0d5adc61862 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:18.194570 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:38:18.202336 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd7ff625-b37e-4d9c-9ce8-c0d5adc61862 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:18.207545 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a97e00be-c91b-420e-aed4-ee56277d0725 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:18.208227 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a97e00be-c91b-420e-aed4-ee56277d0725 admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:18.208811 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a97e00be-c91b-420e-aed4-ee56277d0725 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:18.209083 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a97e00be-c91b-420e-aed4-ee56277d0725 admin admin] PATH_INFO: `/v3/projects/9aaa942d093a4b70976fdd342e16a17b/users/ddbcee24fcdc46c684343de16898ff62/roles/c51fe7c5ad804040879acfb54269455f` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:18.211040 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a97e00be-c91b-420e-aed4-ee56277d0725 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9aaa942d093a4b70976fdd342e16a17b, user_id=ddbcee24fcdc46c684343de16898ff62, role_id=c51fe7c5ad804040879acfb54269455f)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:18.213732 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a97e00be-c91b-420e-aed4-ee56277d0725 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:18.224463 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1110/2196] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:38:18 2022] PUT /identity/v3/projects/9aaa942d093a4b70976fdd342e16a17b/users/ddbcee24fcdc46c684343de16898ff62/roles/c51fe7c5ad804040879acfb54269455f => generated 0 bytes in 36 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:38:18.236037 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c734aabf-41a3-4a6c-8bca-d82e38361d7d None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:18.236327 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c734aabf-41a3-4a6c-8bca-d82e38361d7d None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:18.236602 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c734aabf-41a3-4a6c-8bca-d82e38361d7d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:18.266589 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-c734aabf-41a3-4a6c-8bca-d82e38361d7d None None] MFA Rules not processed for user `ddbcee24fcdc46c684343de16898ff62`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:38:18.267815 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-c734aabf-41a3-4a6c-8bca-d82e38361d7d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:38:18.338952 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1087/2197] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:38:18 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:38:18.350597 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a97e00be-c91b-420e-aed4-ee56277d0725 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:18.352717 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-a97e00be-c91b-420e-aed4-ee56277d0725 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:38:18.416163 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a97e00be-c91b-420e-aed4-ee56277d0725 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:18.421287 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f04a206-e61c-4fd0-b553-b5ef7287e340 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ffbc136c77b84e8c87128f7247f4ec03', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:18.421896 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f04a206-e61c-4fd0-b553-b5ef7287e340 service cinder] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:18.422176 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f04a206-e61c-4fd0-b553-b5ef7287e340 service cinder] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:18.422450 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f04a206-e61c-4fd0-b553-b5ef7287e340 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:18.433472 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f04a206-e61c-4fd0-b553-b5ef7287e340 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:18.435415 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f04a206-e61c-4fd0-b553-b5ef7287e340 service cinder] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:18.464693 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1111/2198] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:38:18 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:38:18.973084 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc18e2c3-de4f-40f2-8423-c53c9b9932f2 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:18.973547 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc18e2c3-de4f-40f2-8423-c53c9b9932f2 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:18.973755 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc18e2c3-de4f-40f2-8423-c53c9b9932f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:18.993679 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-dc18e2c3-de4f-40f2-8423-c53c9b9932f2 None None] MFA Rules not processed for user `2666d4774776498bbdbce539296e76a9`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:38:18.994978 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-dc18e2c3-de4f-40f2-8423-c53c9b9932f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:38:19.051283 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1088/2199] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:38:18 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:38:19.062542 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f04a206-e61c-4fd0-b553-b5ef7287e340 service cinder] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:19.147688 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f04a206-e61c-4fd0-b553-b5ef7287e340 service cinder] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:19.153774 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de2ddcfc-3e14-4579-a807-1ee4be484bcc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:19.154407 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de2ddcfc-3e14-4579-a807-1ee4be484bcc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:19.154683 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de2ddcfc-3e14-4579-a807-1ee4be484bcc admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:19.154967 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de2ddcfc-3e14-4579-a807-1ee4be484bcc admin admin] PATH_INFO: `/v3/users/d9c92669a89e459488de8a6edda33363` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:19.156075 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de2ddcfc-3e14-4579-a807-1ee4be484bcc admin admin] RBAC: Authorizing `identity:delete_user(user_id=d9c92669a89e459488de8a6edda33363)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:19.158012 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de2ddcfc-3e14-4579-a807-1ee4be484bcc admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:19.207824 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-de2ddcfc-3e14-4579-a807-1ee4be484bcc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd9c92669a89e459488de8a6edda33363'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:19.215777 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-de2ddcfc-3e14-4579-a807-1ee4be484bcc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd9c92669a89e459488de8a6edda33363'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:19.229670 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-de2ddcfc-3e14-4579-a807-1ee4be484bcc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd9c92669a89e459488de8a6edda33363'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:19.230498 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-de2ddcfc-3e14-4579-a807-1ee4be484bcc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd9c92669a89e459488de8a6edda33363'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:19.238068 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-de2ddcfc-3e14-4579-a807-1ee4be484bcc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd9c92669a89e459488de8a6edda33363'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:19.243999 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1112/2200] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:38:19 2022] DELETE /identity/v3/users/d9c92669a89e459488de8a6edda33363 => generated 0 bytes in 182 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:38:19.249109 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c52d85f5-7021-4e29-b011-b58f272c998f None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:19.249492 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c52d85f5-7021-4e29-b011-b58f272c998f None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:19.250298 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c52d85f5-7021-4e29-b011-b58f272c998f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:19.270412 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-c52d85f5-7021-4e29-b011-b58f272c998f None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:38:19.271784 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-c52d85f5-7021-4e29-b011-b58f272c998f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:38:19.338823 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1089/2201] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:38:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:38:19.348989 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de2ddcfc-3e14-4579-a807-1ee4be484bcc admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:19.350487 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-de2ddcfc-3e14-4579-a807-1ee4be484bcc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:38:19.365078 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:38:19.413004 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de2ddcfc-3e14-4579-a807-1ee4be484bcc admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:19.418024 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d20c685-f2c0-4d05-a768-a4164b604e1a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:19.418698 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d20c685-f2c0-4d05-a768-a4164b604e1a service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:19.419010 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d20c685-f2c0-4d05-a768-a4164b604e1a service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:19.419285 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d20c685-f2c0-4d05-a768-a4164b604e1a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:19.429109 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d20c685-f2c0-4d05-a768-a4164b604e1a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:19.430475 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d20c685-f2c0-4d05-a768-a4164b604e1a service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:19.441344 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1113/2202] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:38:19 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:38:19.757448 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c52d85f5-7021-4e29-b011-b58f272c998f None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:19.758894 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-c52d85f5-7021-4e29-b011-b58f272c998f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:38:19.775275 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c52d85f5-7021-4e29-b011-b58f272c998f None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:19.783631 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d04f44ff-fa89-40a2-800a-14fe9387a1ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:19.784338 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d04f44ff-fa89-40a2-800a-14fe9387a1ec admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:19.784743 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d04f44ff-fa89-40a2-800a-14fe9387a1ec admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:19.785028 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d04f44ff-fa89-40a2-800a-14fe9387a1ec admin admin] PATH_INFO: `/v3/projects/0620da484fca4c4dab9423b3825abaa8` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:19.786237 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d04f44ff-fa89-40a2-800a-14fe9387a1ec admin admin] RBAC: Authorizing `identity:delete_project(project_id=0620da484fca4c4dab9423b3825abaa8)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:19.788374 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d04f44ff-fa89-40a2-800a-14fe9387a1ec admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:19.805452 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:38:19.851390 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-d04f44ff-fa89-40a2-800a-14fe9387a1ec admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0620da484fca4c4dab9423b3825abaa8', 'request_id': 'req-d04f44ff-fa89-40a2-800a-14fe9387a1ec'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:19.858774 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-d04f44ff-fa89-40a2-800a-14fe9387a1ec admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0620da484fca4c4dab9423b3825abaa8', 'request_id': 'req-d04f44ff-fa89-40a2-800a-14fe9387a1ec'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:19.877965 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-d04f44ff-fa89-40a2-800a-14fe9387a1ec admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0620da484fca4c4dab9423b3825abaa8', 'request_id': 'req-d04f44ff-fa89-40a2-800a-14fe9387a1ec'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:19.890613 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-d04f44ff-fa89-40a2-800a-14fe9387a1ec admin admin] The token cache is being invalidate because project 0620da484fca4c4dab9423b3825abaa8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:38:19.891024 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-d04f44ff-fa89-40a2-800a-14fe9387a1ec admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:19.892615 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-d04f44ff-fa89-40a2-800a-14fe9387a1ec admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:19.894161 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1090/2203] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:38:19 2022] DELETE /identity/v3/projects/0620da484fca4c4dab9423b3825abaa8 => generated 0 bytes in 137 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:38:19.898194 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d20c685-f2c0-4d05-a768-a4164b604e1a service neutron] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:19.910241 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d20c685-f2c0-4d05-a768-a4164b604e1a service neutron] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:19.970507 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b4a1b5e-e45f-4ee6-b881-f81446d3f447 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:19.971137 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b4a1b5e-e45f-4ee6-b881-f81446d3f447 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:19.971415 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b4a1b5e-e45f-4ee6-b881-f81446d3f447 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:19.971684 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b4a1b5e-e45f-4ee6-b881-f81446d3f447 admin admin] PATH_INFO: `/v3/users/ddbcee24fcdc46c684343de16898ff62` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:19.972828 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b4a1b5e-e45f-4ee6-b881-f81446d3f447 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ddbcee24fcdc46c684343de16898ff62)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:19.974720 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b4a1b5e-e45f-4ee6-b881-f81446d3f447 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:20.026129 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-0b4a1b5e-e45f-4ee6-b881-f81446d3f447 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ddbcee24fcdc46c684343de16898ff62'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:20.033873 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-0b4a1b5e-e45f-4ee6-b881-f81446d3f447 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ddbcee24fcdc46c684343de16898ff62'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:20.047179 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-0b4a1b5e-e45f-4ee6-b881-f81446d3f447 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ddbcee24fcdc46c684343de16898ff62'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:20.047979 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-0b4a1b5e-e45f-4ee6-b881-f81446d3f447 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ddbcee24fcdc46c684343de16898ff62'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:20.055191 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-0b4a1b5e-e45f-4ee6-b881-f81446d3f447 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ddbcee24fcdc46c684343de16898ff62'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:20.062137 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1114/2204] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:38:19 2022] DELETE /identity/v3/users/ddbcee24fcdc46c684343de16898ff62 => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 04:38:20.361318 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d04f44ff-fa89-40a2-800a-14fe9387a1ec admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:20.362890 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-d04f44ff-fa89-40a2-800a-14fe9387a1ec admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:38:20.379963 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d04f44ff-fa89-40a2-800a-14fe9387a1ec admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:20.452416 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f3c9604-3293-4dc0-b150-badff80aff54 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:20.453218 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f3c9604-3293-4dc0-b150-badff80aff54 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:20.453587 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f3c9604-3293-4dc0-b150-badff80aff54 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:20.453919 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f3c9604-3293-4dc0-b150-badff80aff54 admin admin] PATH_INFO: `/v3/projects/9aaa942d093a4b70976fdd342e16a17b` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:20.455458 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f3c9604-3293-4dc0-b150-badff80aff54 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9aaa942d093a4b70976fdd342e16a17b)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:20.457525 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f3c9604-3293-4dc0-b150-badff80aff54 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:20.514110 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-4f3c9604-3293-4dc0-b150-badff80aff54 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9aaa942d093a4b70976fdd342e16a17b', 'request_id': 'req-4f3c9604-3293-4dc0-b150-badff80aff54'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:20.521963 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-4f3c9604-3293-4dc0-b150-badff80aff54 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9aaa942d093a4b70976fdd342e16a17b', 'request_id': 'req-4f3c9604-3293-4dc0-b150-badff80aff54'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:20.528819 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-4f3c9604-3293-4dc0-b150-badff80aff54 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9aaa942d093a4b70976fdd342e16a17b', 'request_id': 'req-4f3c9604-3293-4dc0-b150-badff80aff54'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:20.543162 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-4f3c9604-3293-4dc0-b150-badff80aff54 admin admin] The token cache is being invalidate because project 9aaa942d093a4b70976fdd342e16a17b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:38:20.543463 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-4f3c9604-3293-4dc0-b150-badff80aff54 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:20.544130 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-4f3c9604-3293-4dc0-b150-badff80aff54 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:20.545897 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1091/2205] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:38:20 2022] DELETE /identity/v3/projects/9aaa942d093a4b70976fdd342e16a17b => generated 0 bytes in 185 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:38:20.551489 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1437b995-951d-46f6-801c-8c2786eedacc None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:20.551957 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1437b995-951d-46f6-801c-8c2786eedacc None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:20.552038 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1437b995-951d-46f6-801c-8c2786eedacc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:20.578212 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-1437b995-951d-46f6-801c-8c2786eedacc None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:38:20.579556 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-1437b995-951d-46f6-801c-8c2786eedacc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:38:20.611197 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:38:20.648774 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1115/2206] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:38:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 04:38:20.656003 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-083eb0d9-b2b4-410a-988b-bff3274a2e2b None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:20.656285 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-083eb0d9-b2b4-410a-988b-bff3274a2e2b None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:20.656557 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-083eb0d9-b2b4-410a-988b-bff3274a2e2b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:20.677188 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-083eb0d9-b2b4-410a-988b-bff3274a2e2b None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:38:20.678377 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-083eb0d9-b2b4-410a-988b-bff3274a2e2b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:38:20.687604 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1092/2207] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:38:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:38:20.691683 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1437b995-951d-46f6-801c-8c2786eedacc None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:20.701445 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1437b995-951d-46f6-801c-8c2786eedacc None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:20.706771 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c1ba445-85c9-4dae-9d79-bc1193ee2ef8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:20.707371 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c1ba445-85c9-4dae-9d79-bc1193ee2ef8 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:20.707650 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c1ba445-85c9-4dae-9d79-bc1193ee2ef8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:20.707917 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c1ba445-85c9-4dae-9d79-bc1193ee2ef8 admin admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:20.708857 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c1ba445-85c9-4dae-9d79-bc1193ee2ef8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:38:20.709160 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c1ba445-85c9-4dae-9d79-bc1193ee2ef8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:20.710730 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c1ba445-85c9-4dae-9d79-bc1193ee2ef8 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:20.723505 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1116/2208] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Sep 25 04:38:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:38:20.727712 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-083eb0d9-b2b4-410a-988b-bff3274a2e2b None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:20.738766 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-083eb0d9-b2b4-410a-988b-bff3274a2e2b None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:20.743485 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e322994-a63e-4305-9399-2c999e68354d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:20.744087 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e322994-a63e-4305-9399-2c999e68354d admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:20.744458 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e322994-a63e-4305-9399-2c999e68354d admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:20.744936 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e322994-a63e-4305-9399-2c999e68354d admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:20.745579 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e322994-a63e-4305-9399-2c999e68354d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:20.747510 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e322994-a63e-4305-9399-2c999e68354d admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:20.774195 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1093/2209] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:38:20 2022] POST /identity/v3/projects => generated 361 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:38:20.778849 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c1ba445-85c9-4dae-9d79-bc1193ee2ef8 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:20.789315 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c1ba445-85c9-4dae-9d79-bc1193ee2ef8 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:20.845777 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14805c8d-9e4d-4d91-8e06-1e8ff67d8475 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:20.846384 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14805c8d-9e4d-4d91-8e06-1e8ff67d8475 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:20.846666 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14805c8d-9e4d-4d91-8e06-1e8ff67d8475 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:20.846942 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14805c8d-9e4d-4d91-8e06-1e8ff67d8475 admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:20.847568 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14805c8d-9e4d-4d91-8e06-1e8ff67d8475 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:20.849544 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14805c8d-9e4d-4d91-8e06-1e8ff67d8475 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:20.872131 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1117/2210] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:38:20 2022] POST /identity/v3/users => generated 281 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:38:20.876214 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e322994-a63e-4305-9399-2c999e68354d admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:20.883892 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=83,keystone:DELETE=14,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:38:20.889767 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e322994-a63e-4305-9399-2c999e68354d admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:20.896357 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d8327d2-5968-4324-9344-d4cf9ce6c0d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:20.897007 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d8327d2-5968-4324-9344-d4cf9ce6c0d1 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:20.897291 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d8327d2-5968-4324-9344-d4cf9ce6c0d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:20.897578 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d8327d2-5968-4324-9344-d4cf9ce6c0d1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:20.898217 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d8327d2-5968-4324-9344-d4cf9ce6c0d1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:20.899819 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d8327d2-5968-4324-9344-d4cf9ce6c0d1 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:20.911193 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1094/2211] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:38:20 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:38:20.915234 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14805c8d-9e4d-4d91-8e06-1e8ff67d8475 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:20.926338 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14805c8d-9e4d-4d91-8e06-1e8ff67d8475 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:20.931189 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-592880a3-a11d-43d6-91a4-578ca43ed1e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:20.931904 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-592880a3-a11d-43d6-91a4-578ca43ed1e7 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:20.932986 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-592880a3-a11d-43d6-91a4-578ca43ed1e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:20.933281 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-592880a3-a11d-43d6-91a4-578ca43ed1e7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:20.933959 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-592880a3-a11d-43d6-91a4-578ca43ed1e7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:20.935588 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-592880a3-a11d-43d6-91a4-578ca43ed1e7 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:20.948235 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1118/2212] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:38:20 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:38:20.952762 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d8327d2-5968-4324-9344-d4cf9ce6c0d1 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:20.963054 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d8327d2-5968-4324-9344-d4cf9ce6c0d1 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:20.967870 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff411ac3-4ed6-4f0c-b8ae-b3fafdf1e267 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:20.968521 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff411ac3-4ed6-4f0c-b8ae-b3fafdf1e267 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:20.968833 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff411ac3-4ed6-4f0c-b8ae-b3fafdf1e267 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:20.969107 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff411ac3-4ed6-4f0c-b8ae-b3fafdf1e267 admin admin] PATH_INFO: `/v3/projects/c75106efd7d7454f816d6809714ab284/users/31bded4727034d6baaca809e36ce056a/roles/8f6cc574056c4278bc9778f14dc4b411` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:20.984480 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff411ac3-4ed6-4f0c-b8ae-b3fafdf1e267 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c75106efd7d7454f816d6809714ab284, user_id=31bded4727034d6baaca809e36ce056a, role_id=8f6cc574056c4278bc9778f14dc4b411)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:20.987377 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff411ac3-4ed6-4f0c-b8ae-b3fafdf1e267 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:21.001088 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1095/2213] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:38:20 2022] PUT /identity/v3/projects/c75106efd7d7454f816d6809714ab284/users/31bded4727034d6baaca809e36ce056a/roles/8f6cc574056c4278bc9778f14dc4b411 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:38:21.011942 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04cc5fa3-82d6-448b-9da6-f6d200071ab4 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:21.012231 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04cc5fa3-82d6-448b-9da6-f6d200071ab4 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:21.012502 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04cc5fa3-82d6-448b-9da6-f6d200071ab4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:21.041883 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-04cc5fa3-82d6-448b-9da6-f6d200071ab4 None None] MFA Rules not processed for user `31bded4727034d6baaca809e36ce056a`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:38:21.043372 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-04cc5fa3-82d6-448b-9da6-f6d200071ab4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:38:21.116374 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1119/2214] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:38:21 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:38:21.152686 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff411ac3-4ed6-4f0c-b8ae-b3fafdf1e267 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:21.154640 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-ff411ac3-4ed6-4f0c-b8ae-b3fafdf1e267 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:38:21.226492 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff411ac3-4ed6-4f0c-b8ae-b3fafdf1e267 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:21.231676 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2e17abe-ac14-407b-8059-86c3bf81de44 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ffbc136c77b84e8c87128f7247f4ec03', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:21.232406 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e17abe-ac14-407b-8059-86c3bf81de44 service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:21.232802 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e17abe-ac14-407b-8059-86c3bf81de44 service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:21.233141 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e17abe-ac14-407b-8059-86c3bf81de44 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:21.245157 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2e17abe-ac14-407b-8059-86c3bf81de44 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:21.246993 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2e17abe-ac14-407b-8059-86c3bf81de44 service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:21.273685 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1096/2215] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:38:21 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 25 04:38:21.277278 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04cc5fa3-82d6-448b-9da6-f6d200071ab4 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:21.325563 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:38:21.361888 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04cc5fa3-82d6-448b-9da6-f6d200071ab4 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:21.366521 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-508449c0-1db7-46c9-85c7-5d75c6a9b78b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:21.367151 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-508449c0-1db7-46c9-85c7-5d75c6a9b78b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:21.367457 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-508449c0-1db7-46c9-85c7-5d75c6a9b78b admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:21.367745 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-508449c0-1db7-46c9-85c7-5d75c6a9b78b admin admin] PATH_INFO: `/v3/users/31bded4727034d6baaca809e36ce056a` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:21.369369 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-508449c0-1db7-46c9-85c7-5d75c6a9b78b admin admin] RBAC: Authorizing `identity:delete_user(user_id=31bded4727034d6baaca809e36ce056a)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:21.372004 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-508449c0-1db7-46c9-85c7-5d75c6a9b78b admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:21.429146 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-508449c0-1db7-46c9-85c7-5d75c6a9b78b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '31bded4727034d6baaca809e36ce056a'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:21.437431 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-508449c0-1db7-46c9-85c7-5d75c6a9b78b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '31bded4727034d6baaca809e36ce056a'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:21.451623 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-508449c0-1db7-46c9-85c7-5d75c6a9b78b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '31bded4727034d6baaca809e36ce056a'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:21.452353 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-508449c0-1db7-46c9-85c7-5d75c6a9b78b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '31bded4727034d6baaca809e36ce056a'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:21.459300 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-508449c0-1db7-46c9-85c7-5d75c6a9b78b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '31bded4727034d6baaca809e36ce056a'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:21.466684 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1120/2216] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:38:21 2022] DELETE /identity/v3/users/31bded4727034d6baaca809e36ce056a => generated 0 bytes in 189 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:38:21.470510 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce7f36e9-7773-468c-b83f-7373411884aa None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:21.470963 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce7f36e9-7773-468c-b83f-7373411884aa None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:21.471056 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce7f36e9-7773-468c-b83f-7373411884aa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:21.490896 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-ce7f36e9-7773-468c-b83f-7373411884aa None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:38:21.492070 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-ce7f36e9-7773-468c-b83f-7373411884aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:38:21.563254 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1097/2217] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:38:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:38:21.574755 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-508449c0-1db7-46c9-85c7-5d75c6a9b78b admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:21.577587 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-508449c0-1db7-46c9-85c7-5d75c6a9b78b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:38:21.657271 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-508449c0-1db7-46c9-85c7-5d75c6a9b78b admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:21.662552 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d12e8d27-8735-41f2-a029-8187eceea789 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:21.663172 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d12e8d27-8735-41f2-a029-8187eceea789 service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:21.663455 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d12e8d27-8735-41f2-a029-8187eceea789 service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:21.663724 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d12e8d27-8735-41f2-a029-8187eceea789 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:21.674704 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d12e8d27-8735-41f2-a029-8187eceea789 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:21.676202 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d12e8d27-8735-41f2-a029-8187eceea789 service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:21.687333 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1121/2218] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:38:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 114 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 25 04:38:22.165811 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce7f36e9-7773-468c-b83f-7373411884aa None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:22.167289 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-ce7f36e9-7773-468c-b83f-7373411884aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:38:22.179167 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce7f36e9-7773-468c-b83f-7373411884aa None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:22.184846 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e40d6b55-6baa-4fb1-8594-f07aae1552ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:22.185496 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e40d6b55-6baa-4fb1-8594-f07aae1552ee admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:22.185774 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e40d6b55-6baa-4fb1-8594-f07aae1552ee admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:22.186047 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e40d6b55-6baa-4fb1-8594-f07aae1552ee admin admin] PATH_INFO: `/v3/projects/c75106efd7d7454f816d6809714ab284` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:22.187904 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e40d6b55-6baa-4fb1-8594-f07aae1552ee admin admin] RBAC: Authorizing `identity:delete_project(project_id=c75106efd7d7454f816d6809714ab284)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:22.190143 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e40d6b55-6baa-4fb1-8594-f07aae1552ee admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:22.248900 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-e40d6b55-6baa-4fb1-8594-f07aae1552ee admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c75106efd7d7454f816d6809714ab284', 'request_id': 'req-e40d6b55-6baa-4fb1-8594-f07aae1552ee'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:22.257347 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-e40d6b55-6baa-4fb1-8594-f07aae1552ee admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c75106efd7d7454f816d6809714ab284', 'request_id': 'req-e40d6b55-6baa-4fb1-8594-f07aae1552ee'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:22.263777 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-e40d6b55-6baa-4fb1-8594-f07aae1552ee admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c75106efd7d7454f816d6809714ab284', 'request_id': 'req-e40d6b55-6baa-4fb1-8594-f07aae1552ee'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:22.265967 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:38:22.277504 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-e40d6b55-6baa-4fb1-8594-f07aae1552ee admin admin] The token cache is being invalidate because project c75106efd7d7454f816d6809714ab284 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:38:22.278258 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-e40d6b55-6baa-4fb1-8594-f07aae1552ee admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:22.279032 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-e40d6b55-6baa-4fb1-8594-f07aae1552ee admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:22.281102 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1098/2219] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:38:22 2022] DELETE /identity/v3/projects/c75106efd7d7454f816d6809714ab284 => generated 0 bytes in 116 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 04:38:22.285716 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccad1936-0279-47a2-b82a-d630b4e3b28a None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:22.286001 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccad1936-0279-47a2-b82a-d630b4e3b28a None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:22.286478 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccad1936-0279-47a2-b82a-d630b4e3b28a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:22.307416 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-ccad1936-0279-47a2-b82a-d630b4e3b28a None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:38:22.309117 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-ccad1936-0279-47a2-b82a-d630b4e3b28a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:38:22.365836 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:38:22.391591 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1122/2220] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:38:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 04:38:22.401281 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-421858c1-4242-4a43-8231-56d8217c08e7 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:22.401668 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-421858c1-4242-4a43-8231-56d8217c08e7 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:22.401967 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-421858c1-4242-4a43-8231-56d8217c08e7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:22.425372 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-421858c1-4242-4a43-8231-56d8217c08e7 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:38:22.426594 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-421858c1-4242-4a43-8231-56d8217c08e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:38:22.436049 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1099/2221] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:38:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:38:22.440045 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccad1936-0279-47a2-b82a-d630b4e3b28a None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:22.452484 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccad1936-0279-47a2-b82a-d630b4e3b28a None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:22.457556 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecfdfa46-d6c1-4c5d-93b4-86cafc78179a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:22.458176 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecfdfa46-d6c1-4c5d-93b4-86cafc78179a admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:22.458725 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecfdfa46-d6c1-4c5d-93b4-86cafc78179a admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:22.459017 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecfdfa46-d6c1-4c5d-93b4-86cafc78179a admin admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:22.459736 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecfdfa46-d6c1-4c5d-93b4-86cafc78179a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:38:22.460070 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecfdfa46-d6c1-4c5d-93b4-86cafc78179a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:22.461700 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecfdfa46-d6c1-4c5d-93b4-86cafc78179a admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:22.475732 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1123/2222] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Sep 25 04:38:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:38:22.480198 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-421858c1-4242-4a43-8231-56d8217c08e7 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:22.490018 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-421858c1-4242-4a43-8231-56d8217c08e7 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:22.496587 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcaa6eb1-f4f3-4dc7-acd4-cba1b830dfe1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:22.497250 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcaa6eb1-f4f3-4dc7-acd4-cba1b830dfe1 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:22.497530 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcaa6eb1-f4f3-4dc7-acd4-cba1b830dfe1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:22.497980 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcaa6eb1-f4f3-4dc7-acd4-cba1b830dfe1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:22.498638 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcaa6eb1-f4f3-4dc7-acd4-cba1b830dfe1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:22.500619 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcaa6eb1-f4f3-4dc7-acd4-cba1b830dfe1 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:22.533955 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1100/2223] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:38:22 2022] POST /identity/v3/projects => generated 375 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:38:22.535245 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecfdfa46-d6c1-4c5d-93b4-86cafc78179a admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:22.546314 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecfdfa46-d6c1-4c5d-93b4-86cafc78179a admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:22.612535 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-757f58a2-ad05-48af-9677-85dba7421739 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:22.613236 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-757f58a2-ad05-48af-9677-85dba7421739 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:22.613451 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-757f58a2-ad05-48af-9677-85dba7421739 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:22.613719 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-757f58a2-ad05-48af-9677-85dba7421739 admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:22.614478 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-757f58a2-ad05-48af-9677-85dba7421739 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:22.616525 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-757f58a2-ad05-48af-9677-85dba7421739 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:22.645020 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1124/2224] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:38:22 2022] POST /identity/v3/users => generated 288 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:38:22.646168 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcaa6eb1-f4f3-4dc7-acd4-cba1b830dfe1 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:22.658074 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcaa6eb1-f4f3-4dc7-acd4-cba1b830dfe1 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:22.663069 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4807996b-7bef-48b5-a72d-2689bda01969 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:22.663759 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4807996b-7bef-48b5-a72d-2689bda01969 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:22.664114 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4807996b-7bef-48b5-a72d-2689bda01969 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:22.664463 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4807996b-7bef-48b5-a72d-2689bda01969 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:22.665219 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4807996b-7bef-48b5-a72d-2689bda01969 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:22.667114 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4807996b-7bef-48b5-a72d-2689bda01969 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:22.678835 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1101/2225] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:38:22 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:38:22.682649 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-757f58a2-ad05-48af-9677-85dba7421739 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:22.694016 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-757f58a2-ad05-48af-9677-85dba7421739 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:22.699333 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b59f863a-e0f4-4899-82db-a5bd51b9c42c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:22.699939 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b59f863a-e0f4-4899-82db-a5bd51b9c42c admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:22.700220 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b59f863a-e0f4-4899-82db-a5bd51b9c42c admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:22.700493 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b59f863a-e0f4-4899-82db-a5bd51b9c42c admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:22.701183 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b59f863a-e0f4-4899-82db-a5bd51b9c42c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:22.702940 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b59f863a-e0f4-4899-82db-a5bd51b9c42c admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:22.717195 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1125/2226] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:38:22 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:38:22.721420 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4807996b-7bef-48b5-a72d-2689bda01969 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:22.732314 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4807996b-7bef-48b5-a72d-2689bda01969 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:22.737469 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8606fbbf-528f-4a6c-8668-68385e9340b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:22.738224 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8606fbbf-528f-4a6c-8668-68385e9340b2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:22.738573 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8606fbbf-528f-4a6c-8668-68385e9340b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:22.738916 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8606fbbf-528f-4a6c-8668-68385e9340b2 admin admin] PATH_INFO: `/v3/projects/052bdcb6269b4dc493a88b538031c1ad/users/9d95c89ff5b04606b7f62f0dfc1ce1f8/roles/8f6cc574056c4278bc9778f14dc4b411` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:22.754727 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8606fbbf-528f-4a6c-8668-68385e9340b2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=052bdcb6269b4dc493a88b538031c1ad, user_id=9d95c89ff5b04606b7f62f0dfc1ce1f8, role_id=8f6cc574056c4278bc9778f14dc4b411)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:22.757472 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8606fbbf-528f-4a6c-8668-68385e9340b2 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:22.771137 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1102/2227] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:38:22 2022] PUT /identity/v3/projects/052bdcb6269b4dc493a88b538031c1ad/users/9d95c89ff5b04606b7f62f0dfc1ce1f8/roles/8f6cc574056c4278bc9778f14dc4b411 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 04:38:22.782184 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e945c8a8-003b-40c8-86a7-f59327d81b96 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:22.782537 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e945c8a8-003b-40c8-86a7-f59327d81b96 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:22.782836 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e945c8a8-003b-40c8-86a7-f59327d81b96 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:22.814295 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-e945c8a8-003b-40c8-86a7-f59327d81b96 None None] MFA Rules not processed for user `9d95c89ff5b04606b7f62f0dfc1ce1f8`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:38:22.815536 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-e945c8a8-003b-40c8-86a7-f59327d81b96 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:38:22.891481 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1126/2228] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:38:22 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:38:22.895787 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8606fbbf-528f-4a6c-8668-68385e9340b2 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:22.963502 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8606fbbf-528f-4a6c-8668-68385e9340b2 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:22.968393 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcc8e3cd-29f4-4e38-aaff-04243c09582b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:22.969056 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcc8e3cd-29f4-4e38-aaff-04243c09582b admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:22.969337 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcc8e3cd-29f4-4e38-aaff-04243c09582b admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:22.970184 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcc8e3cd-29f4-4e38-aaff-04243c09582b admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:22.970291 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcc8e3cd-29f4-4e38-aaff-04243c09582b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:22.972164 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcc8e3cd-29f4-4e38-aaff-04243c09582b admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:23.004488 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1103/2229] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:38:22 2022] POST /identity/v3/projects => generated 373 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:38:23.005755 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e945c8a8-003b-40c8-86a7-f59327d81b96 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:23.017697 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e945c8a8-003b-40c8-86a7-f59327d81b96 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:23.067001 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:38:23.082103 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a758b5e8-bf1f-4721-b1b4-57bb7cc95a05 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:23.082876 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a758b5e8-bf1f-4721-b1b4-57bb7cc95a05 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:23.083157 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a758b5e8-bf1f-4721-b1b4-57bb7cc95a05 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:23.083682 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a758b5e8-bf1f-4721-b1b4-57bb7cc95a05 admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:23.084324 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a758b5e8-bf1f-4721-b1b4-57bb7cc95a05 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:23.086337 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a758b5e8-bf1f-4721-b1b4-57bb7cc95a05 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:23.111291 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1127/2230] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:38:23 2022] POST /identity/v3/users => generated 293 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:38:23.115218 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcc8e3cd-29f4-4e38-aaff-04243c09582b admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:23.128662 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcc8e3cd-29f4-4e38-aaff-04243c09582b admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:23.135131 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fc993c5-7e25-4cb3-a1ad-46ef04efe6e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:23.136878 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fc993c5-7e25-4cb3-a1ad-46ef04efe6e7 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:23.137170 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fc993c5-7e25-4cb3-a1ad-46ef04efe6e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:23.137521 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fc993c5-7e25-4cb3-a1ad-46ef04efe6e7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:23.138443 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fc993c5-7e25-4cb3-a1ad-46ef04efe6e7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:23.140278 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fc993c5-7e25-4cb3-a1ad-46ef04efe6e7 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:23.156396 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1104/2231] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:38:23 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:38:23.158027 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a758b5e8-bf1f-4721-b1b4-57bb7cc95a05 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:23.171371 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a758b5e8-bf1f-4721-b1b4-57bb7cc95a05 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:23.177035 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad80eecf-0c9a-48c4-87e0-f9f7500f767b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:23.178203 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad80eecf-0c9a-48c4-87e0-f9f7500f767b admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:23.178624 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad80eecf-0c9a-48c4-87e0-f9f7500f767b admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:23.178982 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad80eecf-0c9a-48c4-87e0-f9f7500f767b admin admin] PATH_INFO: `/v3/projects/342fab146d354470b4256c3f561321aa/users/3bc28c6fab694eab91b4ad6b6d93df04/roles/c51fe7c5ad804040879acfb54269455f` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:23.198795 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad80eecf-0c9a-48c4-87e0-f9f7500f767b admin admin] RBAC: Authorizing `identity:create_grant(project_id=342fab146d354470b4256c3f561321aa, user_id=3bc28c6fab694eab91b4ad6b6d93df04, role_id=c51fe7c5ad804040879acfb54269455f)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:23.201504 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad80eecf-0c9a-48c4-87e0-f9f7500f767b admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:23.214709 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1128/2232] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:38:23 2022] PUT /identity/v3/projects/342fab146d354470b4256c3f561321aa/users/3bc28c6fab694eab91b4ad6b6d93df04/roles/c51fe7c5ad804040879acfb54269455f => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:38:23.218638 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fc993c5-7e25-4cb3-a1ad-46ef04efe6e7 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:23.228595 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fc993c5-7e25-4cb3-a1ad-46ef04efe6e7 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:23.289446 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53ee290a-6bb3-4676-b4dc-11b4475a93de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:23.290306 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53ee290a-6bb3-4676-b4dc-11b4475a93de admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:23.290585 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53ee290a-6bb3-4676-b4dc-11b4475a93de admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:23.290857 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53ee290a-6bb3-4676-b4dc-11b4475a93de admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:23.292218 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53ee290a-6bb3-4676-b4dc-11b4475a93de admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:23.294104 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53ee290a-6bb3-4676-b4dc-11b4475a93de admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:23.308350 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1105/2233] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:38:23 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:38:23.309658 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad80eecf-0c9a-48c4-87e0-f9f7500f767b admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:23.325464 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad80eecf-0c9a-48c4-87e0-f9f7500f767b admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:23.334719 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-108ccd62-9189-4cc5-b4d7-1db6297059ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:23.335421 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-108ccd62-9189-4cc5-b4d7-1db6297059ec admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:23.335704 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-108ccd62-9189-4cc5-b4d7-1db6297059ec admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:23.335981 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-108ccd62-9189-4cc5-b4d7-1db6297059ec admin admin] PATH_INFO: `/v3/projects/342fab146d354470b4256c3f561321aa/users/3bc28c6fab694eab91b4ad6b6d93df04/roles/c51fe7c5ad804040879acfb54269455f` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:23.338015 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-108ccd62-9189-4cc5-b4d7-1db6297059ec admin admin] RBAC: Authorizing `identity:create_grant(project_id=342fab146d354470b4256c3f561321aa, user_id=3bc28c6fab694eab91b4ad6b6d93df04, role_id=c51fe7c5ad804040879acfb54269455f)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:23.340730 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-108ccd62-9189-4cc5-b4d7-1db6297059ec admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:23.351407 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1129/2234] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:38:23 2022] PUT /identity/v3/projects/342fab146d354470b4256c3f561321aa/users/3bc28c6fab694eab91b4ad6b6d93df04/roles/c51fe7c5ad804040879acfb54269455f => generated 0 bytes in 42 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:38:23.362257 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f524c49-d71a-445f-8a78-8fc8760b3121 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:23.362566 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f524c49-d71a-445f-8a78-8fc8760b3121 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:23.362828 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f524c49-d71a-445f-8a78-8fc8760b3121 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:23.370643 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:38:23.397821 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-3f524c49-d71a-445f-8a78-8fc8760b3121 None None] MFA Rules not processed for user `3bc28c6fab694eab91b4ad6b6d93df04`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:38:23.398974 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-3f524c49-d71a-445f-8a78-8fc8760b3121 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:38:23.487640 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1106/2235] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:38:23 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:38:23.501683 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-108ccd62-9189-4cc5-b4d7-1db6297059ec admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:23.503733 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-108ccd62-9189-4cc5-b4d7-1db6297059ec admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:38:23.569621 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-108ccd62-9189-4cc5-b4d7-1db6297059ec admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:23.574059 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbeda6dd-2062-4749-832a-dec59dc18010 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ffbc136c77b84e8c87128f7247f4ec03', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:23.574663 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbeda6dd-2062-4749-832a-dec59dc18010 service cinder] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:23.574940 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbeda6dd-2062-4749-832a-dec59dc18010 service cinder] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:23.575226 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbeda6dd-2062-4749-832a-dec59dc18010 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:23.632959 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbeda6dd-2062-4749-832a-dec59dc18010 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:23.634420 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbeda6dd-2062-4749-832a-dec59dc18010 service cinder] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:23.643889 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1130/2236] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:38:23 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 143 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 25 04:38:24.872111 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f524c49-d71a-445f-8a78-8fc8760b3121 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:24.873619 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-3f524c49-d71a-445f-8a78-8fc8760b3121 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:38:24.930253 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f524c49-d71a-445f-8a78-8fc8760b3121 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:24.936456 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a2c3fd0-4741-4aea-bb28-8e6f7140ce5d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:24.937097 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a2c3fd0-4741-4aea-bb28-8e6f7140ce5d service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:24.937380 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a2c3fd0-4741-4aea-bb28-8e6f7140ce5d service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:24.937655 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a2c3fd0-4741-4aea-bb28-8e6f7140ce5d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:24.939416 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-6a2c3fd0-4741-4aea-bb28-8e6f7140ce5d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:38:25.005945 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a2c3fd0-4741-4aea-bb28-8e6f7140ce5d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:25.007468 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a2c3fd0-4741-4aea-bb28-8e6f7140ce5d service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:25.020268 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1107/2237] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:38:24 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 149 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:38:26.228313 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbeda6dd-2062-4749-832a-dec59dc18010 service cinder] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:26.230758 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-dbeda6dd-2062-4749-832a-dec59dc18010 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:38:26.290483 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:38:26.305715 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbeda6dd-2062-4749-832a-dec59dc18010 service cinder] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:26.310660 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca31a835-4ff7-41d2-a718-02bf8140adc7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:26.311296 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca31a835-4ff7-41d2-a718-02bf8140adc7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:26.311571 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca31a835-4ff7-41d2-a718-02bf8140adc7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:26.311839 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca31a835-4ff7-41d2-a718-02bf8140adc7 admin admin] PATH_INFO: `/v3/users/08499712078f400a8bca5da2742137ec` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:26.312983 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca31a835-4ff7-41d2-a718-02bf8140adc7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=08499712078f400a8bca5da2742137ec)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:26.314901 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca31a835-4ff7-41d2-a718-02bf8140adc7 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:26.368363 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-ca31a835-4ff7-41d2-a718-02bf8140adc7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '08499712078f400a8bca5da2742137ec'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:26.376866 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-ca31a835-4ff7-41d2-a718-02bf8140adc7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '08499712078f400a8bca5da2742137ec'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:26.392116 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-ca31a835-4ff7-41d2-a718-02bf8140adc7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '08499712078f400a8bca5da2742137ec'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:26.392839 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-ca31a835-4ff7-41d2-a718-02bf8140adc7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '08499712078f400a8bca5da2742137ec'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:26.400137 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-ca31a835-4ff7-41d2-a718-02bf8140adc7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '08499712078f400a8bca5da2742137ec'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:26.406870 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1131/2238] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:38:26 2022] DELETE /identity/v3/users/08499712078f400a8bca5da2742137ec => generated 0 bytes in 179 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:38:26.411348 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d888c492-8263-4c74-b6e2-671364b010e4 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:26.411847 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d888c492-8263-4c74-b6e2-671364b010e4 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:26.412095 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d888c492-8263-4c74-b6e2-671364b010e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:26.432067 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-d888c492-8263-4c74-b6e2-671364b010e4 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:38:26.433590 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-d888c492-8263-4c74-b6e2-671364b010e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:38:26.469387 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:38:26.502748 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1108/2239] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:38:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:38:26.513775 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca31a835-4ff7-41d2-a718-02bf8140adc7 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:26.515353 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-ca31a835-4ff7-41d2-a718-02bf8140adc7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:38:26.580993 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca31a835-4ff7-41d2-a718-02bf8140adc7 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:26.587388 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c2b54fc-7e44-484f-b627-a0bce413b000 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:26.588713 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c2b54fc-7e44-484f-b627-a0bce413b000 service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:26.589060 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c2b54fc-7e44-484f-b627-a0bce413b000 service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:26.589334 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c2b54fc-7e44-484f-b627-a0bce413b000 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:26.601948 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c2b54fc-7e44-484f-b627-a0bce413b000 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:26.603494 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c2b54fc-7e44-484f-b627-a0bce413b000 service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:26.613856 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1132/2240] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:38:26 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 25 04:38:26.925857 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d888c492-8263-4c74-b6e2-671364b010e4 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:26.927367 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-d888c492-8263-4c74-b6e2-671364b010e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:38:26.949554 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d888c492-8263-4c74-b6e2-671364b010e4 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:26.955290 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-affe3212-3105-45f6-b6c0-a8f8fe362db8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:26.956286 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-affe3212-3105-45f6-b6c0-a8f8fe362db8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:26.956578 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-affe3212-3105-45f6-b6c0-a8f8fe362db8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:26.956893 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-affe3212-3105-45f6-b6c0-a8f8fe362db8 admin admin] PATH_INFO: `/v3/projects/099c5ea5a055473186e6e9e76edb9d3e` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:26.958089 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-affe3212-3105-45f6-b6c0-a8f8fe362db8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=099c5ea5a055473186e6e9e76edb9d3e)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:26.960658 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-affe3212-3105-45f6-b6c0-a8f8fe362db8 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:27.018202 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-affe3212-3105-45f6-b6c0-a8f8fe362db8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '099c5ea5a055473186e6e9e76edb9d3e', 'request_id': 'req-affe3212-3105-45f6-b6c0-a8f8fe362db8'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:27.027255 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-affe3212-3105-45f6-b6c0-a8f8fe362db8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '099c5ea5a055473186e6e9e76edb9d3e', 'request_id': 'req-affe3212-3105-45f6-b6c0-a8f8fe362db8'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:27.041899 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-affe3212-3105-45f6-b6c0-a8f8fe362db8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '099c5ea5a055473186e6e9e76edb9d3e', 'request_id': 'req-affe3212-3105-45f6-b6c0-a8f8fe362db8'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:27.053496 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-affe3212-3105-45f6-b6c0-a8f8fe362db8 admin admin] The token cache is being invalidate because project 099c5ea5a055473186e6e9e76edb9d3e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:38:27.053789 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-affe3212-3105-45f6-b6c0-a8f8fe362db8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:27.054446 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-affe3212-3105-45f6-b6c0-a8f8fe362db8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:27.055941 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1109/2241] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:38:26 2022] DELETE /identity/v3/projects/099c5ea5a055473186e6e9e76edb9d3e => generated 0 bytes in 131 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:38:27.061302 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46fa802e-f61a-4d98-969a-2b45b5634c28 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:27.061668 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46fa802e-f61a-4d98-969a-2b45b5634c28 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:27.062492 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46fa802e-f61a-4d98-969a-2b45b5634c28 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:27.082085 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-46fa802e-f61a-4d98-969a-2b45b5634c28 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:38:27.083397 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-46fa802e-f61a-4d98-969a-2b45b5634c28 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:38:27.146902 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1133/2242] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:38:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:38:27.155113 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ab6fd0d-6e58-4956-8f6d-8cc308b92483 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:27.155403 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ab6fd0d-6e58-4956-8f6d-8cc308b92483 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:27.155677 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ab6fd0d-6e58-4956-8f6d-8cc308b92483 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:27.176768 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-2ab6fd0d-6e58-4956-8f6d-8cc308b92483 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:38:27.178003 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-2ab6fd0d-6e58-4956-8f6d-8cc308b92483 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:38:27.188130 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1110/2243] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:38:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:38:27.192122 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46fa802e-f61a-4d98-969a-2b45b5634c28 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:27.203920 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46fa802e-f61a-4d98-969a-2b45b5634c28 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:27.208897 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4837ec0b-7fb2-456b-96ba-a3d8d734c737 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:27.209514 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4837ec0b-7fb2-456b-96ba-a3d8d734c737 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:27.209798 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4837ec0b-7fb2-456b-96ba-a3d8d734c737 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:27.210070 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4837ec0b-7fb2-456b-96ba-a3d8d734c737 admin admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:27.211153 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4837ec0b-7fb2-456b-96ba-a3d8d734c737 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:38:27.211492 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4837ec0b-7fb2-456b-96ba-a3d8d734c737 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:27.213136 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4837ec0b-7fb2-456b-96ba-a3d8d734c737 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:27.225375 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1134/2244] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Sep 25 04:38:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:38:27.230295 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ab6fd0d-6e58-4956-8f6d-8cc308b92483 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:27.239580 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ab6fd0d-6e58-4956-8f6d-8cc308b92483 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:27.245096 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24b4237c-b151-466e-a762-4cae26db9a1e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:27.246023 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24b4237c-b151-466e-a762-4cae26db9a1e admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:27.246287 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24b4237c-b151-466e-a762-4cae26db9a1e admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:27.246555 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24b4237c-b151-466e-a762-4cae26db9a1e admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:27.247196 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24b4237c-b151-466e-a762-4cae26db9a1e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:27.249135 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24b4237c-b151-466e-a762-4cae26db9a1e admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:27.278895 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1111/2245] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:38:27 2022] POST /identity/v3/projects => generated 373 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:38:27.283366 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4837ec0b-7fb2-456b-96ba-a3d8d734c737 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:27.288549 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:38:27.295447 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4837ec0b-7fb2-456b-96ba-a3d8d734c737 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:27.300790 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24b4237c-b151-466e-a762-4cae26db9a1e admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:27.302937 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-24b4237c-b151-466e-a762-4cae26db9a1e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:38:27.367730 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70e97a50-094a-456d-9cb7-704f1012946c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:27.368733 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70e97a50-094a-456d-9cb7-704f1012946c admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:27.369016 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70e97a50-094a-456d-9cb7-704f1012946c admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:27.369334 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70e97a50-094a-456d-9cb7-704f1012946c admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:27.370989 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70e97a50-094a-456d-9cb7-704f1012946c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:27.373198 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70e97a50-094a-456d-9cb7-704f1012946c admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:27.378671 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24b4237c-b151-466e-a762-4cae26db9a1e admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:27.383575 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-259425f1-509a-4bb8-892b-9b808f2cd8ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:27.384220 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-259425f1-509a-4bb8-892b-9b808f2cd8ce admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:27.384504 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-259425f1-509a-4bb8-892b-9b808f2cd8ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:27.384814 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-259425f1-509a-4bb8-892b-9b808f2cd8ce admin admin] PATH_INFO: `/v3/users/9d95c89ff5b04606b7f62f0dfc1ce1f8` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:27.386351 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-259425f1-509a-4bb8-892b-9b808f2cd8ce admin admin] RBAC: Authorizing `identity:delete_user(user_id=9d95c89ff5b04606b7f62f0dfc1ce1f8)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:27.388624 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-259425f1-509a-4bb8-892b-9b808f2cd8ce admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:27.397503 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1135/2246] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:38:27 2022] POST /identity/v3/users => generated 287 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 04:38:27.399453 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70e97a50-094a-456d-9cb7-704f1012946c admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:27.411053 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70e97a50-094a-456d-9cb7-704f1012946c admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:27.418315 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52d2dff0-1244-462c-9fbb-c1cd927327d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:27.419059 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52d2dff0-1244-462c-9fbb-c1cd927327d3 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:27.419464 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52d2dff0-1244-462c-9fbb-c1cd927327d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:27.419741 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52d2dff0-1244-462c-9fbb-c1cd927327d3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:27.420389 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52d2dff0-1244-462c-9fbb-c1cd927327d3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:27.422028 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52d2dff0-1244-462c-9fbb-c1cd927327d3 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:27.433891 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:38:27.438461 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1136/2247] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:38:27 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:38:27.439810 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52d2dff0-1244-462c-9fbb-c1cd927327d3 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:27.447634 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-259425f1-509a-4bb8-892b-9b808f2cd8ce admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9d95c89ff5b04606b7f62f0dfc1ce1f8'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:27.451065 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52d2dff0-1244-462c-9fbb-c1cd927327d3 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:27.455843 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-259425f1-509a-4bb8-892b-9b808f2cd8ce admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9d95c89ff5b04606b7f62f0dfc1ce1f8'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:27.457677 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a93816b-066c-4298-a914-5852efc2f06a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:27.458908 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a93816b-066c-4298-a914-5852efc2f06a admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:27.459193 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a93816b-066c-4298-a914-5852efc2f06a admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:27.459463 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a93816b-066c-4298-a914-5852efc2f06a admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:27.460107 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a93816b-066c-4298-a914-5852efc2f06a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:27.461757 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a93816b-066c-4298-a914-5852efc2f06a admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:27.473097 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1137/2248] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:38:27 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:38:27.476898 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a93816b-066c-4298-a914-5852efc2f06a admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:27.482320 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-259425f1-509a-4bb8-892b-9b808f2cd8ce admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9d95c89ff5b04606b7f62f0dfc1ce1f8'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:27.483471 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-259425f1-509a-4bb8-892b-9b808f2cd8ce admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9d95c89ff5b04606b7f62f0dfc1ce1f8'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:27.489871 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a93816b-066c-4298-a914-5852efc2f06a admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:27.491449 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-259425f1-509a-4bb8-892b-9b808f2cd8ce admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9d95c89ff5b04606b7f62f0dfc1ce1f8'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:27.495531 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc7ebe02-bd3c-4bd5-8b34-5e3bbef05116 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:27.496270 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc7ebe02-bd3c-4bd5-8b34-5e3bbef05116 admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:27.496618 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc7ebe02-bd3c-4bd5-8b34-5e3bbef05116 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:27.496987 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc7ebe02-bd3c-4bd5-8b34-5e3bbef05116 admin admin] PATH_INFO: `/v3/projects/49d74820c72648a88b03dace38305c37/users/18c96c44f5b043df9dd73c4cd5eb1d47/roles/8f6cc574056c4278bc9778f14dc4b411` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:27.499176 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1112/2249] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:38:27 2022] DELETE /identity/v3/users/9d95c89ff5b04606b7f62f0dfc1ce1f8 => generated 0 bytes in 199 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 04:38:27.503782 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-596808a8-70e9-4f46-be52-70a720cf3036 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:27.504120 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-596808a8-70e9-4f46-be52-70a720cf3036 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:27.504672 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-596808a8-70e9-4f46-be52-70a720cf3036 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:27.517502 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc7ebe02-bd3c-4bd5-8b34-5e3bbef05116 admin admin] RBAC: Authorizing `identity:create_grant(project_id=49d74820c72648a88b03dace38305c37, user_id=18c96c44f5b043df9dd73c4cd5eb1d47, role_id=8f6cc574056c4278bc9778f14dc4b411)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:27.520881 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc7ebe02-bd3c-4bd5-8b34-5e3bbef05116 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:27.524341 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-596808a8-70e9-4f46-be52-70a720cf3036 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:38:27.526620 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-596808a8-70e9-4f46-be52-70a720cf3036 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:38:27.537258 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1138/2250] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:38:27 2022] PUT /identity/v3/projects/49d74820c72648a88b03dace38305c37/users/18c96c44f5b043df9dd73c4cd5eb1d47/roles/8f6cc574056c4278bc9778f14dc4b411 => generated 0 bytes in 60 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:38:27.550024 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24511390-ec3d-4783-a10f-25c6602b0b89 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:27.550389 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24511390-ec3d-4783-a10f-25c6602b0b89 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:27.550637 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24511390-ec3d-4783-a10f-25c6602b0b89 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:27.582080 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-24511390-ec3d-4783-a10f-25c6602b0b89 None None] MFA Rules not processed for user `18c96c44f5b043df9dd73c4cd5eb1d47`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:38:27.583977 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-24511390-ec3d-4783-a10f-25c6602b0b89 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:38:27.607447 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1113/2251] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:38:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:38:27.614488 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-596808a8-70e9-4f46-be52-70a720cf3036 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:27.616065 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-596808a8-70e9-4f46-be52-70a720cf3036 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:38:27.662670 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1139/2252] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:38:27 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:38:27.674000 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24511390-ec3d-4783-a10f-25c6602b0b89 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:27.675417 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-24511390-ec3d-4783-a10f-25c6602b0b89 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:38:27.679884 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-596808a8-70e9-4f46-be52-70a720cf3036 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:27.719853 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:38:27.735981 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91842570-7296-4ee3-92a9-54fd8e11add0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:27.736610 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91842570-7296-4ee3-92a9-54fd8e11add0 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:27.736929 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91842570-7296-4ee3-92a9-54fd8e11add0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:27.737346 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91842570-7296-4ee3-92a9-54fd8e11add0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:27.739762 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24511390-ec3d-4783-a10f-25c6602b0b89 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:27.743594 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-853b2aa8-a594-43d1-9ac7-d4223b7dffa0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ffbc136c77b84e8c87128f7247f4ec03', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:27.744763 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-853b2aa8-a594-43d1-9ac7-d4223b7dffa0 service cinder] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:27.745062 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-853b2aa8-a594-43d1-9ac7-d4223b7dffa0 service cinder] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:27.745335 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-853b2aa8-a594-43d1-9ac7-d4223b7dffa0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:27.754582 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-853b2aa8-a594-43d1-9ac7-d4223b7dffa0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:27.756033 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-853b2aa8-a594-43d1-9ac7-d4223b7dffa0 service cinder] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:27.764336 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:38:27.769549 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1140/2253] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:38:27 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 25 04:38:27.813026 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91842570-7296-4ee3-92a9-54fd8e11add0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:27.814590 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91842570-7296-4ee3-92a9-54fd8e11add0 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:27.825938 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1114/2254] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:38:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 212 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 25 04:38:28.178387 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-853b2aa8-a594-43d1-9ac7-d4223b7dffa0 service cinder] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:28.180364 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-853b2aa8-a594-43d1-9ac7-d4223b7dffa0 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:38:28.208962 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-853b2aa8-a594-43d1-9ac7-d4223b7dffa0 service cinder] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:28.215450 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bf6c36e-86be-4362-975d-86d7c890bcd6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:28.225409 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bf6c36e-86be-4362-975d-86d7c890bcd6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:28.225876 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bf6c36e-86be-4362-975d-86d7c890bcd6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:28.226366 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bf6c36e-86be-4362-975d-86d7c890bcd6 admin admin] PATH_INFO: `/v3/projects/052bdcb6269b4dc493a88b538031c1ad` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:28.228474 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bf6c36e-86be-4362-975d-86d7c890bcd6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=052bdcb6269b4dc493a88b538031c1ad)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:28.233575 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bf6c36e-86be-4362-975d-86d7c890bcd6 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:28.293148 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-4bf6c36e-86be-4362-975d-86d7c890bcd6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '052bdcb6269b4dc493a88b538031c1ad', 'request_id': 'req-4bf6c36e-86be-4362-975d-86d7c890bcd6'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:28.301706 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-4bf6c36e-86be-4362-975d-86d7c890bcd6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '052bdcb6269b4dc493a88b538031c1ad', 'request_id': 'req-4bf6c36e-86be-4362-975d-86d7c890bcd6'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:28.307950 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-4bf6c36e-86be-4362-975d-86d7c890bcd6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '052bdcb6269b4dc493a88b538031c1ad', 'request_id': 'req-4bf6c36e-86be-4362-975d-86d7c890bcd6'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:28.319235 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-4bf6c36e-86be-4362-975d-86d7c890bcd6 admin admin] The token cache is being invalidate because project 052bdcb6269b4dc493a88b538031c1ad was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81434) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:38:28.319529 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-4bf6c36e-86be-4362-975d-86d7c890bcd6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:28.320265 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-4bf6c36e-86be-4362-975d-86d7c890bcd6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:28.321792 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1141/2255] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:38:28 2022] DELETE /identity/v3/projects/052bdcb6269b4dc493a88b538031c1ad => generated 0 bytes in 144 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:38:28.326483 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91842570-7296-4ee3-92a9-54fd8e11add0 service neutron] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:28.336409 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91842570-7296-4ee3-92a9-54fd8e11add0 service neutron] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:28.392912 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a1dec4c-6da0-4f98-b663-972ef00c2d96 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:28.393543 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a1dec4c-6da0-4f98-b663-972ef00c2d96 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:28.393828 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a1dec4c-6da0-4f98-b663-972ef00c2d96 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:28.394102 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a1dec4c-6da0-4f98-b663-972ef00c2d96 admin admin] PATH_INFO: `/v3/users/3bc28c6fab694eab91b4ad6b6d93df04` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:28.395222 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a1dec4c-6da0-4f98-b663-972ef00c2d96 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3bc28c6fab694eab91b4ad6b6d93df04)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:28.397201 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a1dec4c-6da0-4f98-b663-972ef00c2d96 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:28.442914 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-2a1dec4c-6da0-4f98-b663-972ef00c2d96 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3bc28c6fab694eab91b4ad6b6d93df04'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:28.451248 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-2a1dec4c-6da0-4f98-b663-972ef00c2d96 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3bc28c6fab694eab91b4ad6b6d93df04'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:28.463945 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-2a1dec4c-6da0-4f98-b663-972ef00c2d96 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3bc28c6fab694eab91b4ad6b6d93df04'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:28.464613 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-2a1dec4c-6da0-4f98-b663-972ef00c2d96 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3bc28c6fab694eab91b4ad6b6d93df04'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:28.471764 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-2a1dec4c-6da0-4f98-b663-972ef00c2d96 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3bc28c6fab694eab91b4ad6b6d93df04'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:28.477806 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1115/2256] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:38:28 2022] DELETE /identity/v3/users/3bc28c6fab694eab91b4ad6b6d93df04 => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:38:28.759796 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bf6c36e-86be-4362-975d-86d7c890bcd6 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:28.761431 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-4bf6c36e-86be-4362-975d-86d7c890bcd6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:38:28.774074 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bf6c36e-86be-4362-975d-86d7c890bcd6 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:28.832417 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ca8e041-398e-48e9-8164-cd0e998055dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:28.833094 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca8e041-398e-48e9-8164-cd0e998055dc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:28.833400 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca8e041-398e-48e9-8164-cd0e998055dc admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:28.833926 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca8e041-398e-48e9-8164-cd0e998055dc admin admin] PATH_INFO: `/v3/projects/342fab146d354470b4256c3f561321aa` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:28.835038 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ca8e041-398e-48e9-8164-cd0e998055dc admin admin] RBAC: Authorizing `identity:delete_project(project_id=342fab146d354470b4256c3f561321aa)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:28.836972 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ca8e041-398e-48e9-8164-cd0e998055dc admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:28.884417 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-5ca8e041-398e-48e9-8164-cd0e998055dc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '342fab146d354470b4256c3f561321aa', 'request_id': 'req-5ca8e041-398e-48e9-8164-cd0e998055dc'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:28.892381 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-5ca8e041-398e-48e9-8164-cd0e998055dc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '342fab146d354470b4256c3f561321aa', 'request_id': 'req-5ca8e041-398e-48e9-8164-cd0e998055dc'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:28.897199 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:38:28.898467 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-5ca8e041-398e-48e9-8164-cd0e998055dc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '342fab146d354470b4256c3f561321aa', 'request_id': 'req-5ca8e041-398e-48e9-8164-cd0e998055dc'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:28.911634 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-5ca8e041-398e-48e9-8164-cd0e998055dc admin admin] The token cache is being invalidate because project 342fab146d354470b4256c3f561321aa was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81434) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:38:28.911923 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-5ca8e041-398e-48e9-8164-cd0e998055dc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:28.912553 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-5ca8e041-398e-48e9-8164-cd0e998055dc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:28.914050 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1142/2257] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:38:28 2022] DELETE /identity/v3/projects/342fab146d354470b4256c3f561321aa => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:38:28.918985 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23e73eb1-c58b-48a1-acd9-11fc5b203ef8 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:28.919276 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23e73eb1-c58b-48a1-acd9-11fc5b203ef8 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:28.919578 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23e73eb1-c58b-48a1-acd9-11fc5b203ef8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:28.939265 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-23e73eb1-c58b-48a1-acd9-11fc5b203ef8 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:38:28.940468 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-23e73eb1-c58b-48a1-acd9-11fc5b203ef8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:38:28.973346 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:38:29.001666 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1116/2258] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:38:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:38:29.009517 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f854c932-598e-4cc6-a9d8-38372ec99e1d None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:29.009822 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f854c932-598e-4cc6-a9d8-38372ec99e1d None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:29.010670 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f854c932-598e-4cc6-a9d8-38372ec99e1d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:29.030295 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-f854c932-598e-4cc6-a9d8-38372ec99e1d None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:38:29.031505 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-f854c932-598e-4cc6-a9d8-38372ec99e1d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:38:29.039463 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1143/2259] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:38:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:38:29.043616 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23e73eb1-c58b-48a1-acd9-11fc5b203ef8 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:29.053332 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23e73eb1-c58b-48a1-acd9-11fc5b203ef8 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:29.058529 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d20cf01e-7bad-4bf2-a6cf-cc012bca1347 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:29.059128 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d20cf01e-7bad-4bf2-a6cf-cc012bca1347 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:29.059410 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d20cf01e-7bad-4bf2-a6cf-cc012bca1347 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:29.059685 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d20cf01e-7bad-4bf2-a6cf-cc012bca1347 admin admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:29.060333 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d20cf01e-7bad-4bf2-a6cf-cc012bca1347 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:38:29.060692 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d20cf01e-7bad-4bf2-a6cf-cc012bca1347 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:29.062291 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d20cf01e-7bad-4bf2-a6cf-cc012bca1347 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:29.074314 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1117/2260] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Sep 25 04:38:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:38:29.078331 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f854c932-598e-4cc6-a9d8-38372ec99e1d None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:29.087724 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f854c932-598e-4cc6-a9d8-38372ec99e1d None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:29.092559 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb8b0ddd-04cd-467d-b5fd-a68f1488e3b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:29.093197 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb8b0ddd-04cd-467d-b5fd-a68f1488e3b2 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:29.093481 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb8b0ddd-04cd-467d-b5fd-a68f1488e3b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:29.093757 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb8b0ddd-04cd-467d-b5fd-a68f1488e3b2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:29.094398 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb8b0ddd-04cd-467d-b5fd-a68f1488e3b2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:29.096281 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb8b0ddd-04cd-467d-b5fd-a68f1488e3b2 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:29.125355 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1144/2261] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:38:29 2022] POST /identity/v3/projects => generated 373 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:38:29.129806 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d20cf01e-7bad-4bf2-a6cf-cc012bca1347 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:29.140386 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d20cf01e-7bad-4bf2-a6cf-cc012bca1347 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:29.202188 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fc30d1f-ae00-43e3-967e-994041853a2c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:29.202815 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fc30d1f-ae00-43e3-967e-994041853a2c admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:29.203097 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fc30d1f-ae00-43e3-967e-994041853a2c admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:29.204463 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fc30d1f-ae00-43e3-967e-994041853a2c admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:29.205279 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fc30d1f-ae00-43e3-967e-994041853a2c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:29.207212 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fc30d1f-ae00-43e3-967e-994041853a2c admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:29.233341 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1118/2262] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:38:29 2022] POST /identity/v3/users => generated 287 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:38:29.234481 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb8b0ddd-04cd-467d-b5fd-a68f1488e3b2 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:29.247405 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb8b0ddd-04cd-467d-b5fd-a68f1488e3b2 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:29.252861 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e37d40d0-68a3-4107-bf95-d3851bef370a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:29.253553 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e37d40d0-68a3-4107-bf95-d3851bef370a admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:29.253838 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e37d40d0-68a3-4107-bf95-d3851bef370a admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:29.254208 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e37d40d0-68a3-4107-bf95-d3851bef370a admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:29.254861 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e37d40d0-68a3-4107-bf95-d3851bef370a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:29.256558 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e37d40d0-68a3-4107-bf95-d3851bef370a admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:29.272799 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1145/2263] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:38:29 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:38:29.273185 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fc30d1f-ae00-43e3-967e-994041853a2c admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:29.283483 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fc30d1f-ae00-43e3-967e-994041853a2c admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:29.288608 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b19fc0ae-ad86-4377-a1e7-bbccb0767795 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:29.289287 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b19fc0ae-ad86-4377-a1e7-bbccb0767795 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:29.289579 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b19fc0ae-ad86-4377-a1e7-bbccb0767795 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:29.289859 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b19fc0ae-ad86-4377-a1e7-bbccb0767795 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:29.290759 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b19fc0ae-ad86-4377-a1e7-bbccb0767795 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:29.293470 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b19fc0ae-ad86-4377-a1e7-bbccb0767795 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:29.311126 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1119/2264] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:38:29 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 25 04:38:29.313900 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e37d40d0-68a3-4107-bf95-d3851bef370a admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:29.325241 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e37d40d0-68a3-4107-bf95-d3851bef370a admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:29.331776 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00fc832a-862c-4ebc-adb6-baf6f806c15c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:29.332447 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00fc832a-862c-4ebc-adb6-baf6f806c15c admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:29.332761 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00fc832a-862c-4ebc-adb6-baf6f806c15c admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:29.333044 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00fc832a-862c-4ebc-adb6-baf6f806c15c admin admin] PATH_INFO: `/v3/projects/b7d504f39d52488abf8282bc0ec2786c/users/10fac643eaad4a29913b3090c8931d54/roles/8f6cc574056c4278bc9778f14dc4b411` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:29.351657 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00fc832a-862c-4ebc-adb6-baf6f806c15c admin admin] RBAC: Authorizing `identity:create_grant(project_id=b7d504f39d52488abf8282bc0ec2786c, user_id=10fac643eaad4a29913b3090c8931d54, role_id=8f6cc574056c4278bc9778f14dc4b411)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:29.355046 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00fc832a-862c-4ebc-adb6-baf6f806c15c admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:29.378262 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1146/2265] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:38:29 2022] PUT /identity/v3/projects/b7d504f39d52488abf8282bc0ec2786c/users/10fac643eaad4a29913b3090c8931d54/roles/8f6cc574056c4278bc9778f14dc4b411 => generated 0 bytes in 65 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:38:29.389886 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7619aa28-e4a7-4def-a019-fe00f53776e8 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:29.390173 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7619aa28-e4a7-4def-a019-fe00f53776e8 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:29.390451 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7619aa28-e4a7-4def-a019-fe00f53776e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:29.422679 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-7619aa28-e4a7-4def-a019-fe00f53776e8 None None] MFA Rules not processed for user `10fac643eaad4a29913b3090c8931d54`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:38:29.425109 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-7619aa28-e4a7-4def-a019-fe00f53776e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:38:29.498964 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1120/2266] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:38:29 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 04:38:29.509480 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00fc832a-862c-4ebc-adb6-baf6f806c15c admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:29.511584 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-00fc832a-862c-4ebc-adb6-baf6f806c15c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:38:29.587247 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00fc832a-862c-4ebc-adb6-baf6f806c15c admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:29.592214 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7550499-1d42-4de9-ae6c-bd9ab59c462d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ffbc136c77b84e8c87128f7247f4ec03', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:29.593872 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7550499-1d42-4de9-ae6c-bd9ab59c462d service cinder] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:29.594242 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7550499-1d42-4de9-ae6c-bd9ab59c462d service cinder] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:29.594627 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7550499-1d42-4de9-ae6c-bd9ab59c462d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:29.606635 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7550499-1d42-4de9-ae6c-bd9ab59c462d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:29.608803 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7550499-1d42-4de9-ae6c-bd9ab59c462d service cinder] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:29.620408 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1147/2267] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:38:29 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:38:32.102264 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe5e55b3-a93c-4b0b-86b4-b922793788b8 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:32.102739 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe5e55b3-a93c-4b0b-86b4-b922793788b8 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:32.102739 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe5e55b3-a93c-4b0b-86b4-b922793788b8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:32.122695 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-fe5e55b3-a93c-4b0b-86b4-b922793788b8 None None] MFA Rules not processed for user `10fac643eaad4a29913b3090c8931d54`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:38:32.123920 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-fe5e55b3-a93c-4b0b-86b4-b922793788b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:38:32.130762 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1121/2268] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:38:32 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:38:39.270245 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7550499-1d42-4de9-ae6c-bd9ab59c462d service cinder] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:39.273043 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-c7550499-1d42-4de9-ae6c-bd9ab59c462d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:38:39.349146 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7550499-1d42-4de9-ae6c-bd9ab59c462d service cinder] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:39.353716 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc94bd05-34bf-491f-90a0-3ede6b1d9917 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:39.354345 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc94bd05-34bf-491f-90a0-3ede6b1d9917 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:39.354623 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc94bd05-34bf-491f-90a0-3ede6b1d9917 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:39.354894 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc94bd05-34bf-491f-90a0-3ede6b1d9917 admin admin] PATH_INFO: `/v3/users/18c96c44f5b043df9dd73c4cd5eb1d47` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:39.356038 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc94bd05-34bf-491f-90a0-3ede6b1d9917 admin admin] RBAC: Authorizing `identity:delete_user(user_id=18c96c44f5b043df9dd73c4cd5eb1d47)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:39.357988 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc94bd05-34bf-491f-90a0-3ede6b1d9917 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:39.393066 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=93,keystone:INSERT=2,keystone:DELETE=5 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:38:39.406968 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-bc94bd05-34bf-491f-90a0-3ede6b1d9917 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '18c96c44f5b043df9dd73c4cd5eb1d47'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:39.414045 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-bc94bd05-34bf-491f-90a0-3ede6b1d9917 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '18c96c44f5b043df9dd73c4cd5eb1d47'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:39.427187 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-bc94bd05-34bf-491f-90a0-3ede6b1d9917 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '18c96c44f5b043df9dd73c4cd5eb1d47'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:39.427871 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-bc94bd05-34bf-491f-90a0-3ede6b1d9917 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '18c96c44f5b043df9dd73c4cd5eb1d47'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:39.435746 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-bc94bd05-34bf-491f-90a0-3ede6b1d9917 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '18c96c44f5b043df9dd73c4cd5eb1d47'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:39.445068 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1148/2269] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:38:39 2022] DELETE /identity/v3/users/18c96c44f5b043df9dd73c4cd5eb1d47 => generated 0 bytes in 175 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:38:39.453614 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7987f4a-7303-4f3e-b98d-41b29185133d None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:39.454126 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7987f4a-7303-4f3e-b98d-41b29185133d None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:39.454286 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7987f4a-7303-4f3e-b98d-41b29185133d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:39.475619 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-e7987f4a-7303-4f3e-b98d-41b29185133d None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:38:39.477087 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-e7987f4a-7303-4f3e-b98d-41b29185133d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:38:39.498197 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:38:39.537964 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1122/2270] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:38:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:38:39.548210 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc94bd05-34bf-491f-90a0-3ede6b1d9917 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:39.549671 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-bc94bd05-34bf-491f-90a0-3ede6b1d9917 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:38:39.609511 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc94bd05-34bf-491f-90a0-3ede6b1d9917 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:39.613240 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61356a11-d53e-4547-bb69-e80bf71e4b3c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:39.613853 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61356a11-d53e-4547-bb69-e80bf71e4b3c service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:39.614141 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61356a11-d53e-4547-bb69-e80bf71e4b3c service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:39.614416 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61356a11-d53e-4547-bb69-e80bf71e4b3c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:39.625325 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61356a11-d53e-4547-bb69-e80bf71e4b3c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:39.626777 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61356a11-d53e-4547-bb69-e80bf71e4b3c service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:39.637434 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1149/2271] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:38:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:38:40.011550 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7987f4a-7303-4f3e-b98d-41b29185133d None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:40.013058 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-e7987f4a-7303-4f3e-b98d-41b29185133d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:38:40.029084 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7987f4a-7303-4f3e-b98d-41b29185133d None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:40.034426 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad285864-0827-4b11-a005-3fb67e7dde3b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:40.035070 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad285864-0827-4b11-a005-3fb67e7dde3b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:40.035400 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad285864-0827-4b11-a005-3fb67e7dde3b admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:40.035614 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad285864-0827-4b11-a005-3fb67e7dde3b admin admin] PATH_INFO: `/v3/projects/49d74820c72648a88b03dace38305c37` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:40.036774 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad285864-0827-4b11-a005-3fb67e7dde3b admin admin] RBAC: Authorizing `identity:delete_project(project_id=49d74820c72648a88b03dace38305c37)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:40.038691 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad285864-0827-4b11-a005-3fb67e7dde3b admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:40.087829 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-ad285864-0827-4b11-a005-3fb67e7dde3b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '49d74820c72648a88b03dace38305c37', 'request_id': 'req-ad285864-0827-4b11-a005-3fb67e7dde3b'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:40.095565 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-ad285864-0827-4b11-a005-3fb67e7dde3b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '49d74820c72648a88b03dace38305c37', 'request_id': 'req-ad285864-0827-4b11-a005-3fb67e7dde3b'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:40.101716 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-ad285864-0827-4b11-a005-3fb67e7dde3b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '49d74820c72648a88b03dace38305c37', 'request_id': 'req-ad285864-0827-4b11-a005-3fb67e7dde3b'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:40.113329 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-ad285864-0827-4b11-a005-3fb67e7dde3b admin admin] The token cache is being invalidate because project 49d74820c72648a88b03dace38305c37 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:38:40.113621 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-ad285864-0827-4b11-a005-3fb67e7dde3b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:40.114272 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-ad285864-0827-4b11-a005-3fb67e7dde3b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:38:40.115762 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1123/2272] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:38:40 2022] DELETE /identity/v3/projects/49d74820c72648a88b03dace38305c37 => generated 0 bytes in 105 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 04:38:40.122549 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad6121a6-4b0a-466d-890b-3425191dd692 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:40.122829 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad6121a6-4b0a-466d-890b-3425191dd692 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:40.123120 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad6121a6-4b0a-466d-890b-3425191dd692 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:40.143396 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-ad6121a6-4b0a-466d-890b-3425191dd692 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:38:40.144606 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-ad6121a6-4b0a-466d-890b-3425191dd692 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:38:40.208915 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1150/2273] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:38:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:38:40.217234 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2caa78e-f4c3-45b7-ac67-436bfe6d28e9 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:40.217528 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2caa78e-f4c3-45b7-ac67-436bfe6d28e9 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:40.217805 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2caa78e-f4c3-45b7-ac67-436bfe6d28e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:40.238497 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-a2caa78e-f4c3-45b7-ac67-436bfe6d28e9 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:38:40.239711 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-a2caa78e-f4c3-45b7-ac67-436bfe6d28e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:38:40.248034 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1124/2274] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:38:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:38:40.252222 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad6121a6-4b0a-466d-890b-3425191dd692 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:40.262375 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad6121a6-4b0a-466d-890b-3425191dd692 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:40.268091 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-053947c8-133e-4e5b-a77a-6b249e4878b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:40.268738 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-053947c8-133e-4e5b-a77a-6b249e4878b0 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:40.269023 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-053947c8-133e-4e5b-a77a-6b249e4878b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:40.269301 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-053947c8-133e-4e5b-a77a-6b249e4878b0 admin admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:40.269952 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-053947c8-133e-4e5b-a77a-6b249e4878b0 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:38:40.270255 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-053947c8-133e-4e5b-a77a-6b249e4878b0 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:40.271837 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-053947c8-133e-4e5b-a77a-6b249e4878b0 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:40.284843 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1151/2275] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Sep 25 04:38:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:38:40.289160 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2caa78e-f4c3-45b7-ac67-436bfe6d28e9 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:40.299092 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2caa78e-f4c3-45b7-ac67-436bfe6d28e9 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:40.304708 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-046a7d69-25cf-4511-b827-2ae9fdcef912 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:40.305508 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-046a7d69-25cf-4511-b827-2ae9fdcef912 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:40.305790 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-046a7d69-25cf-4511-b827-2ae9fdcef912 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:40.306197 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-046a7d69-25cf-4511-b827-2ae9fdcef912 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:40.307549 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-046a7d69-25cf-4511-b827-2ae9fdcef912 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:40.309502 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-046a7d69-25cf-4511-b827-2ae9fdcef912 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:40.338372 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1125/2276] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:38:40 2022] POST /identity/v3/projects => generated 383 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:38:40.342570 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-053947c8-133e-4e5b-a77a-6b249e4878b0 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:40.352867 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-053947c8-133e-4e5b-a77a-6b249e4878b0 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:40.391698 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:38:40.412404 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89a08251-81fa-4ea5-9459-df4ef26e898e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:40.413051 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89a08251-81fa-4ea5-9459-df4ef26e898e admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:40.413336 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89a08251-81fa-4ea5-9459-df4ef26e898e admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:40.413611 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89a08251-81fa-4ea5-9459-df4ef26e898e admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:40.414250 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89a08251-81fa-4ea5-9459-df4ef26e898e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:40.416156 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89a08251-81fa-4ea5-9459-df4ef26e898e admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:40.438908 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1152/2277] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:38:40 2022] POST /identity/v3/users => generated 292 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:38:40.442966 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-046a7d69-25cf-4511-b827-2ae9fdcef912 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:40.453259 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-046a7d69-25cf-4511-b827-2ae9fdcef912 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:40.457958 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cd4fd9a-2c3a-46b4-b001-c63deeacd3eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:40.458554 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cd4fd9a-2c3a-46b4-b001-c63deeacd3eb admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:40.459167 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cd4fd9a-2c3a-46b4-b001-c63deeacd3eb admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:40.459367 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cd4fd9a-2c3a-46b4-b001-c63deeacd3eb admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:40.460003 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cd4fd9a-2c3a-46b4-b001-c63deeacd3eb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:40.461664 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cd4fd9a-2c3a-46b4-b001-c63deeacd3eb admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:40.476249 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1126/2278] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:38:40 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:38:40.480286 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89a08251-81fa-4ea5-9459-df4ef26e898e admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:40.490327 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89a08251-81fa-4ea5-9459-df4ef26e898e admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:40.495141 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a956f88e-c329-4b8a-9c3e-6039e515289a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:40.495737 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a956f88e-c329-4b8a-9c3e-6039e515289a admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:40.496018 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a956f88e-c329-4b8a-9c3e-6039e515289a admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:40.496294 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a956f88e-c329-4b8a-9c3e-6039e515289a admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:40.496968 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a956f88e-c329-4b8a-9c3e-6039e515289a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:40.498569 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a956f88e-c329-4b8a-9c3e-6039e515289a admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:40.509419 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1153/2279] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:38:40 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:38:40.514470 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cd4fd9a-2c3a-46b4-b001-c63deeacd3eb admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:40.524902 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cd4fd9a-2c3a-46b4-b001-c63deeacd3eb admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:40.530562 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c21cc6e-5cb7-4103-bc9a-ff4783a6e69c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:40.531227 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c21cc6e-5cb7-4103-bc9a-ff4783a6e69c admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:40.531543 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c21cc6e-5cb7-4103-bc9a-ff4783a6e69c admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:40.531849 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c21cc6e-5cb7-4103-bc9a-ff4783a6e69c admin admin] PATH_INFO: `/v3/projects/b5c1180472fb4243bd45e44914a46918/users/5902801c596048389599b2b18df16dd4/roles/8f6cc574056c4278bc9778f14dc4b411` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:40.546918 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c21cc6e-5cb7-4103-bc9a-ff4783a6e69c admin admin] RBAC: Authorizing `identity:create_grant(project_id=b5c1180472fb4243bd45e44914a46918, user_id=5902801c596048389599b2b18df16dd4, role_id=8f6cc574056c4278bc9778f14dc4b411)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:40.549630 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c21cc6e-5cb7-4103-bc9a-ff4783a6e69c admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:40.562882 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1127/2280] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:38:40 2022] PUT /identity/v3/projects/b5c1180472fb4243bd45e44914a46918/users/5902801c596048389599b2b18df16dd4/roles/8f6cc574056c4278bc9778f14dc4b411 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:38:40.567881 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72cf08c3-c7fd-4bff-8033-a4360f7a2d79 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:40.568187 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72cf08c3-c7fd-4bff-8033-a4360f7a2d79 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:40.568467 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72cf08c3-c7fd-4bff-8033-a4360f7a2d79 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:40.589950 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-72cf08c3-c7fd-4bff-8033-a4360f7a2d79 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:38:40.591164 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-72cf08c3-c7fd-4bff-8033-a4360f7a2d79 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:38:40.652740 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1154/2281] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:38:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:38:40.662486 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c21cc6e-5cb7-4103-bc9a-ff4783a6e69c admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:40.664239 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-5c21cc6e-5cb7-4103-bc9a-ff4783a6e69c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:38:40.717347 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:38:40.724129 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c21cc6e-5cb7-4103-bc9a-ff4783a6e69c admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:40.728919 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25710d74-b824-45e7-87bd-1041fd04487e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:40.729532 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25710d74-b824-45e7-87bd-1041fd04487e service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:40.729808 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25710d74-b824-45e7-87bd-1041fd04487e service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:40.730082 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25710d74-b824-45e7-87bd-1041fd04487e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:40.739602 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25710d74-b824-45e7-87bd-1041fd04487e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:40.741073 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25710d74-b824-45e7-87bd-1041fd04487e service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:40.751078 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1128/2282] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:38:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:38:45.267408 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bf6cfd7-fa44-4a18-a86d-64e08b5641be None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:45.267937 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bf6cfd7-fa44-4a18-a86d-64e08b5641be None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:45.268064 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bf6cfd7-fa44-4a18-a86d-64e08b5641be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:45.296552 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-8bf6cfd7-fa44-4a18-a86d-64e08b5641be None None] MFA Rules not processed for user `5902801c596048389599b2b18df16dd4`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:38:45.297845 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-8bf6cfd7-fa44-4a18-a86d-64e08b5641be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:38:45.368473 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1155/2283] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:38:45 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:38:45.373273 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25710d74-b824-45e7-87bd-1041fd04487e service neutron] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:45.384292 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25710d74-b824-45e7-87bd-1041fd04487e service neutron] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:45.390157 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d71b79fc-5279-44ba-aa6b-c523085a8c08 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:45.390769 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d71b79fc-5279-44ba-aa6b-c523085a8c08 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:45.391045 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d71b79fc-5279-44ba-aa6b-c523085a8c08 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:45.391501 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d71b79fc-5279-44ba-aa6b-c523085a8c08 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:45.392317 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d71b79fc-5279-44ba-aa6b-c523085a8c08 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:45.394257 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d71b79fc-5279-44ba-aa6b-c523085a8c08 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:45.430165 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1129/2284] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:38:45 2022] POST /identity/v3/projects => generated 383 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:38:45.431490 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bf6cfd7-fa44-4a18-a86d-64e08b5641be None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:45.444001 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bf6cfd7-fa44-4a18-a86d-64e08b5641be None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:45.507100 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:38:45.520156 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df5e12e4-f132-4751-8a0c-9c7609d45614 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:45.520927 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df5e12e4-f132-4751-8a0c-9c7609d45614 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:45.521215 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df5e12e4-f132-4751-8a0c-9c7609d45614 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:45.521492 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df5e12e4-f132-4751-8a0c-9c7609d45614 admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:45.522159 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df5e12e4-f132-4751-8a0c-9c7609d45614 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:45.524142 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df5e12e4-f132-4751-8a0c-9c7609d45614 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:45.547340 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1156/2285] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:38:45 2022] POST /identity/v3/users => generated 298 bytes in 116 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:38:45.552462 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d71b79fc-5279-44ba-aa6b-c523085a8c08 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:45.563444 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d71b79fc-5279-44ba-aa6b-c523085a8c08 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:45.568293 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53edd8b5-2211-4fa0-90bf-7ac0e879195b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:45.568939 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53edd8b5-2211-4fa0-90bf-7ac0e879195b admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:45.569225 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53edd8b5-2211-4fa0-90bf-7ac0e879195b admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:45.569514 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53edd8b5-2211-4fa0-90bf-7ac0e879195b admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:45.570164 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53edd8b5-2211-4fa0-90bf-7ac0e879195b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:45.571892 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53edd8b5-2211-4fa0-90bf-7ac0e879195b admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:45.583073 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1130/2286] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:38:45 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:38:45.587536 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df5e12e4-f132-4751-8a0c-9c7609d45614 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:45.598708 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df5e12e4-f132-4751-8a0c-9c7609d45614 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:45.604132 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-177b6202-d01f-4720-bf78-7eafefcd0095 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:45.604856 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-177b6202-d01f-4720-bf78-7eafefcd0095 admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:45.605140 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-177b6202-d01f-4720-bf78-7eafefcd0095 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:45.605414 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-177b6202-d01f-4720-bf78-7eafefcd0095 admin admin] PATH_INFO: `/v3/projects/388ceb38ad354423992a7cc1eae04f44/users/40bb42d222744b17815207ad07c36c2e/roles/c51fe7c5ad804040879acfb54269455f` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:45.624970 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-177b6202-d01f-4720-bf78-7eafefcd0095 admin admin] RBAC: Authorizing `identity:create_grant(project_id=388ceb38ad354423992a7cc1eae04f44, user_id=40bb42d222744b17815207ad07c36c2e, role_id=c51fe7c5ad804040879acfb54269455f)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:45.629284 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-177b6202-d01f-4720-bf78-7eafefcd0095 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:45.651472 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1157/2287] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:38:45 2022] PUT /identity/v3/projects/388ceb38ad354423992a7cc1eae04f44/users/40bb42d222744b17815207ad07c36c2e/roles/c51fe7c5ad804040879acfb54269455f => generated 0 bytes in 64 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:38:45.655455 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53edd8b5-2211-4fa0-90bf-7ac0e879195b admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:45.671371 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53edd8b5-2211-4fa0-90bf-7ac0e879195b admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:45.743809 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec927fac-67d2-4c11-a393-195081d7e0b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:45.744816 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec927fac-67d2-4c11-a393-195081d7e0b5 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:45.745222 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec927fac-67d2-4c11-a393-195081d7e0b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:45.745499 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec927fac-67d2-4c11-a393-195081d7e0b5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:45.746153 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec927fac-67d2-4c11-a393-195081d7e0b5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:45.747766 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec927fac-67d2-4c11-a393-195081d7e0b5 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:45.758722 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1131/2288] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:38:45 2022] GET /identity/v3/roles => generated 1420 bytes in 104 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:38:45.763500 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-177b6202-d01f-4720-bf78-7eafefcd0095 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:45.777101 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-177b6202-d01f-4720-bf78-7eafefcd0095 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:45.782286 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1568f96a-c12f-493f-a52c-6a54a4f4a0f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:45.782994 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1568f96a-c12f-493f-a52c-6a54a4f4a0f5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:45.783280 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1568f96a-c12f-493f-a52c-6a54a4f4a0f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:45.783558 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1568f96a-c12f-493f-a52c-6a54a4f4a0f5 admin admin] PATH_INFO: `/v3/projects/388ceb38ad354423992a7cc1eae04f44/users/40bb42d222744b17815207ad07c36c2e/roles/c51fe7c5ad804040879acfb54269455f` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:45.785607 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1568f96a-c12f-493f-a52c-6a54a4f4a0f5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=388ceb38ad354423992a7cc1eae04f44, user_id=40bb42d222744b17815207ad07c36c2e, role_id=c51fe7c5ad804040879acfb54269455f)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:45.788518 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1568f96a-c12f-493f-a52c-6a54a4f4a0f5 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:45.799739 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1158/2289] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:38:45 2022] PUT /identity/v3/projects/388ceb38ad354423992a7cc1eae04f44/users/40bb42d222744b17815207ad07c36c2e/roles/c51fe7c5ad804040879acfb54269455f => generated 0 bytes in 37 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 04:38:50.749503 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23c2d63a-403d-4309-afde-545860463590 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:50.750090 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23c2d63a-403d-4309-afde-545860463590 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:50.750664 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23c2d63a-403d-4309-afde-545860463590 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:50.781823 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-23c2d63a-403d-4309-afde-545860463590 None None] MFA Rules not processed for user `40bb42d222744b17815207ad07c36c2e`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:38:50.782989 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-23c2d63a-403d-4309-afde-545860463590 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:38:50.854056 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1132/2290] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:38:50 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:38:50.867621 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1568f96a-c12f-493f-a52c-6a54a4f4a0f5 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:50.870196 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-1568f96a-c12f-493f-a52c-6a54a4f4a0f5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:38:50.940046 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1568f96a-c12f-493f-a52c-6a54a4f4a0f5 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:50.944231 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e46621d-2ee8-445e-87fe-dc150109191a service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '316eaa67d5464091af47efd489b96b87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:50.945128 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e46621d-2ee8-445e-87fe-dc150109191a service glance] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:50.945408 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e46621d-2ee8-445e-87fe-dc150109191a service glance] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:50.945678 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e46621d-2ee8-445e-87fe-dc150109191a service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:51.016384 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e46621d-2ee8-445e-87fe-dc150109191a service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:51.018126 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e46621d-2ee8-445e-87fe-dc150109191a service glance] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:51.028713 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1159/2291] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:38:50 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 162 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:38:51.257069 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5b569fa-5458-4831-b9b9-b88c811af7c4 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:51.257365 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5b569fa-5458-4831-b9b9-b88c811af7c4 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:51.257641 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5b569fa-5458-4831-b9b9-b88c811af7c4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:51.280142 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-e5b569fa-5458-4831-b9b9-b88c811af7c4 None None] MFA Rules not processed for user `050b3aab44bb4b17aa4a1ba7cccdb231`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:38:51.282132 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-e5b569fa-5458-4831-b9b9-b88c811af7c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:38:51.341034 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:38:51.371266 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1133/2292] 10.222.0.25 () {58 vars in 1009 bytes} [Sun Sep 25 04:38:51 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:38:51.420116 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e46621d-2ee8-445e-87fe-dc150109191a service glance] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:51.422064 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-5e46621d-2ee8-445e-87fe-dc150109191a service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:38:51.508443 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:38:51.530228 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e46621d-2ee8-445e-87fe-dc150109191a service glance] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:51.535657 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-345533e2-5417-4966-990d-d7b1ebcff104 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '23ea88f46e7740d295148a93ade29dd9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:51.536452 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-345533e2-5417-4966-990d-d7b1ebcff104 service swift] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:51.536774 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-345533e2-5417-4966-990d-d7b1ebcff104 service swift] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:51.537106 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-345533e2-5417-4966-990d-d7b1ebcff104 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:51.552140 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-345533e2-5417-4966-990d-d7b1ebcff104 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:51.553838 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-345533e2-5417-4966-990d-d7b1ebcff104 service swift] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:51.563158 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1160/2293] 10.222.0.25 () {60 vars in 1384 bytes} [Sun Sep 25 04:38:51 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 144 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 25 04:38:55.139595 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5b569fa-5458-4831-b9b9-b88c811af7c4 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:55.143051 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-e5b569fa-5458-4831-b9b9-b88c811af7c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:38:55.296596 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5b569fa-5458-4831-b9b9-b88c811af7c4 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:55.321343 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a42b7e5-7391-472e-9511-ecd51a77bc34 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '676d04ef5a984edcaad63d9156568660', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:55.325378 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a42b7e5-7391-472e-9511-ecd51a77bc34 service placement] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:55.325785 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a42b7e5-7391-472e-9511-ecd51a77bc34 service placement] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:55.326164 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a42b7e5-7391-472e-9511-ecd51a77bc34 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:55.328446 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-9a42b7e5-7391-472e-9511-ecd51a77bc34 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:38:55.346092 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a42b7e5-7391-472e-9511-ecd51a77bc34 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:55.352013 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a42b7e5-7391-472e-9511-ecd51a77bc34 service placement] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:55.387431 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1134/2294] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:38:55 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 248 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:38:57.176684 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeab8c2a-d1a9-4da0-bc59-1c78ed63a126 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:57.177616 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeab8c2a-d1a9-4da0-bc59-1c78ed63a126 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:57.178130 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeab8c2a-d1a9-4da0-bc59-1c78ed63a126 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:57.207353 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-aeab8c2a-d1a9-4da0-bc59-1c78ed63a126 None None] MFA Rules not processed for user `050b3aab44bb4b17aa4a1ba7cccdb231`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:38:57.208834 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-aeab8c2a-d1a9-4da0-bc59-1c78ed63a126 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:38:57.216959 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1161/2295] 10.222.0.25 () {58 vars in 1009 bytes} [Sun Sep 25 04:38:57 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:38:57.234399 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a42b7e5-7391-472e-9511-ecd51a77bc34 service placement] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:57.250746 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a42b7e5-7391-472e-9511-ecd51a77bc34 service placement] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:57.257163 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-804454be-49ac-4bfb-9fe8-f8b2687f8cdc service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '23ea88f46e7740d295148a93ade29dd9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:57.257869 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-804454be-49ac-4bfb-9fe8-f8b2687f8cdc service swift] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:57.258208 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-804454be-49ac-4bfb-9fe8-f8b2687f8cdc service swift] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:57.258556 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-804454be-49ac-4bfb-9fe8-f8b2687f8cdc service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:57.279390 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-804454be-49ac-4bfb-9fe8-f8b2687f8cdc service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:57.281043 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-804454be-49ac-4bfb-9fe8-f8b2687f8cdc service swift] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:57.291823 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1135/2296] 10.222.0.25 () {60 vars in 1384 bytes} [Sun Sep 25 04:38:57 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 58 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 25 04:38:58.035474 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c2089b2-f3f5-46fd-b011-93a719bc17c9 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:58.035764 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c2089b2-f3f5-46fd-b011-93a719bc17c9 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:58.036097 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c2089b2-f3f5-46fd-b011-93a719bc17c9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:58.063384 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-9c2089b2-f3f5-46fd-b011-93a719bc17c9 None None] MFA Rules not processed for user `050b3aab44bb4b17aa4a1ba7cccdb231`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:38:58.065303 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-9c2089b2-f3f5-46fd-b011-93a719bc17c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:38:58.075816 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1162/2297] 10.222.0.25 () {58 vars in 967 bytes} [Sun Sep 25 04:38:58 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:38:58.092357 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-804454be-49ac-4bfb-9fe8-f8b2687f8cdc service swift] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:58.104412 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-804454be-49ac-4bfb-9fe8-f8b2687f8cdc service swift] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:58.109847 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dd2082f-d58c-4fb2-bcc8-f0f9f42eeb04 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '23ea88f46e7740d295148a93ade29dd9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:58.110460 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dd2082f-d58c-4fb2-bcc8-f0f9f42eeb04 service swift] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:58.110747 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dd2082f-d58c-4fb2-bcc8-f0f9f42eeb04 service swift] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:58.111024 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dd2082f-d58c-4fb2-bcc8-f0f9f42eeb04 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:58.123308 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dd2082f-d58c-4fb2-bcc8-f0f9f42eeb04 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:58.124964 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dd2082f-d58c-4fb2-bcc8-f0f9f42eeb04 service swift] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:58.134959 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1136/2298] 10.222.0.25 () {60 vars in 1384 bytes} [Sun Sep 25 04:38:58 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 25 04:38:58.437591 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f041b17d-7361-4fe3-9564-97e77886adc6 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:58.438122 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f041b17d-7361-4fe3-9564-97e77886adc6 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:58.438122 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f041b17d-7361-4fe3-9564-97e77886adc6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:58.458330 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-f041b17d-7361-4fe3-9564-97e77886adc6 None None] MFA Rules not processed for user `050b3aab44bb4b17aa4a1ba7cccdb231`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:38:58.459794 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-f041b17d-7361-4fe3-9564-97e77886adc6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:38:58.480742 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1163/2299] 10.222.0.25 () {58 vars in 1009 bytes} [Sun Sep 25 04:38:58 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:38:58.492081 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dd2082f-d58c-4fb2-bcc8-f0f9f42eeb04 service swift] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:58.504291 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dd2082f-d58c-4fb2-bcc8-f0f9f42eeb04 service swift] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:58.511113 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02a0fe37-2dad-4521-bcf8-2414d38f5d50 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '23ea88f46e7740d295148a93ade29dd9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:58.511762 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02a0fe37-2dad-4521-bcf8-2414d38f5d50 service swift] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:58.512045 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02a0fe37-2dad-4521-bcf8-2414d38f5d50 service swift] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:58.512448 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02a0fe37-2dad-4521-bcf8-2414d38f5d50 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:58.528512 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02a0fe37-2dad-4521-bcf8-2414d38f5d50 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:58.530849 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02a0fe37-2dad-4521-bcf8-2414d38f5d50 service swift] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:58.543294 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1137/2300] 10.222.0.25 () {60 vars in 1384 bytes} [Sun Sep 25 04:38:58 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 52 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 25 04:38:58.969787 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45d0a810-0c29-4621-9412-84e2de32be2b None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:58.970088 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45d0a810-0c29-4621-9412-84e2de32be2b None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:58.970352 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45d0a810-0c29-4621-9412-84e2de32be2b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:58.990500 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-45d0a810-0c29-4621-9412-84e2de32be2b None None] MFA Rules not processed for user `050b3aab44bb4b17aa4a1ba7cccdb231`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:38:58.991711 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-45d0a810-0c29-4621-9412-84e2de32be2b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:38:59.005160 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1164/2301] 10.222.0.25 () {58 vars in 967 bytes} [Sun Sep 25 04:38:58 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:38:59.014713 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02a0fe37-2dad-4521-bcf8-2414d38f5d50 service swift] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:38:59.028726 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02a0fe37-2dad-4521-bcf8-2414d38f5d50 service swift] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:38:59.034721 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3af0efc5-127c-4423-9223-4123c49f2dbe service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '23ea88f46e7740d295148a93ade29dd9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:38:59.035372 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3af0efc5-127c-4423-9223-4123c49f2dbe service swift] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:38:59.035748 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3af0efc5-127c-4423-9223-4123c49f2dbe service swift] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:38:59.036087 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3af0efc5-127c-4423-9223-4123c49f2dbe service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:38:59.046961 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3af0efc5-127c-4423-9223-4123c49f2dbe service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:38:59.048953 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3af0efc5-127c-4423-9223-4123c49f2dbe service swift] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:38:59.059581 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1138/2302] 10.222.0.25 () {60 vars in 1384 bytes} [Sun Sep 25 04:38:59 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 25 04:39:01.519531 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45d0a810-0c29-4621-9412-84e2de32be2b None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:39:01.521010 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-45d0a810-0c29-4621-9412-84e2de32be2b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:39:01.539310 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c4fee1b-f06b-4682-b3e1-17721c60cf2a None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:39:01.539891 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c4fee1b-f06b-4682-b3e1-17721c60cf2a None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:39:01.539891 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c4fee1b-f06b-4682-b3e1-17721c60cf2a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:39:01.559387 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-2c4fee1b-f06b-4682-b3e1-17721c60cf2a None None] MFA Rules not processed for user `5902801c596048389599b2b18df16dd4`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:39:01.560602 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-2c4fee1b-f06b-4682-b3e1-17721c60cf2a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:39:01.567599 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1139/2303] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:39:01 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 04:39:01.596087 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45d0a810-0c29-4621-9412-84e2de32be2b None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:39:01.600813 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b606fb23-e76a-4e80-a624-cef2da936f8b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:39:01.601453 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b606fb23-e76a-4e80-a624-cef2da936f8b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:39:01.602340 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b606fb23-e76a-4e80-a624-cef2da936f8b admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:39:01.602566 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b606fb23-e76a-4e80-a624-cef2da936f8b admin admin] PATH_INFO: `/v3/users/10fac643eaad4a29913b3090c8931d54` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:39:01.604033 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b606fb23-e76a-4e80-a624-cef2da936f8b admin admin] RBAC: Authorizing `identity:delete_user(user_id=10fac643eaad4a29913b3090c8931d54)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:39:01.606027 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b606fb23-e76a-4e80-a624-cef2da936f8b admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:39:01.656822 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-b606fb23-e76a-4e80-a624-cef2da936f8b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '10fac643eaad4a29913b3090c8931d54'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:39:01.664590 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-b606fb23-e76a-4e80-a624-cef2da936f8b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '10fac643eaad4a29913b3090c8931d54'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:39:01.678074 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-b606fb23-e76a-4e80-a624-cef2da936f8b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '10fac643eaad4a29913b3090c8931d54'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:39:01.678768 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-b606fb23-e76a-4e80-a624-cef2da936f8b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '10fac643eaad4a29913b3090c8931d54'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:39:01.686778 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-b606fb23-e76a-4e80-a624-cef2da936f8b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '10fac643eaad4a29913b3090c8931d54'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:39:01.692666 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1165/2304] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:39:01 2022] DELETE /identity/v3/users/10fac643eaad4a29913b3090c8931d54 => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:39:01.697839 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de167403-78c4-4063-85c0-e93c7c719830 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:39:01.698538 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de167403-78c4-4063-85c0-e93c7c719830 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:39:01.698778 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de167403-78c4-4063-85c0-e93c7c719830 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:39:01.720665 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-de167403-78c4-4063-85c0-e93c7c719830 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:39:01.721881 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-de167403-78c4-4063-85c0-e93c7c719830 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:39:01.790817 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1140/2305] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:39:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:39:01.805758 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b606fb23-e76a-4e80-a624-cef2da936f8b admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:39:01.807405 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-b606fb23-e76a-4e80-a624-cef2da936f8b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:39:01.852134 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:39:01.870343 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b606fb23-e76a-4e80-a624-cef2da936f8b admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:39:01.874032 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d34c27d8-58aa-4da6-ae95-e9a30df80862 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:39:01.874642 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d34c27d8-58aa-4da6-ae95-e9a30df80862 service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:39:01.874921 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d34c27d8-58aa-4da6-ae95-e9a30df80862 service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:39:01.875191 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d34c27d8-58aa-4da6-ae95-e9a30df80862 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:39:01.885476 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d34c27d8-58aa-4da6-ae95-e9a30df80862 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:39:01.887041 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d34c27d8-58aa-4da6-ae95-e9a30df80862 service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:39:01.896822 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1166/2306] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:39:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:39:02.071587 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de167403-78c4-4063-85c0-e93c7c719830 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:39:02.073488 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-de167403-78c4-4063-85c0-e93c7c719830 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:39:02.086827 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de167403-78c4-4063-85c0-e93c7c719830 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:39:02.092487 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5919f375-f064-4013-9fdf-25fac9d60f9d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:39:02.093366 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5919f375-f064-4013-9fdf-25fac9d60f9d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:39:02.093653 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5919f375-f064-4013-9fdf-25fac9d60f9d admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:39:02.093933 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5919f375-f064-4013-9fdf-25fac9d60f9d admin admin] PATH_INFO: `/v3/projects/b7d504f39d52488abf8282bc0ec2786c` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:39:02.095209 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5919f375-f064-4013-9fdf-25fac9d60f9d admin admin] RBAC: Authorizing `identity:delete_project(project_id=b7d504f39d52488abf8282bc0ec2786c)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:39:02.097348 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5919f375-f064-4013-9fdf-25fac9d60f9d admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:39:02.107901 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:39:02.148360 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-5919f375-f064-4013-9fdf-25fac9d60f9d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b7d504f39d52488abf8282bc0ec2786c', 'request_id': 'req-5919f375-f064-4013-9fdf-25fac9d60f9d'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:39:02.157222 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-5919f375-f064-4013-9fdf-25fac9d60f9d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b7d504f39d52488abf8282bc0ec2786c', 'request_id': 'req-5919f375-f064-4013-9fdf-25fac9d60f9d'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:39:02.163066 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-5919f375-f064-4013-9fdf-25fac9d60f9d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b7d504f39d52488abf8282bc0ec2786c', 'request_id': 'req-5919f375-f064-4013-9fdf-25fac9d60f9d'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:39:02.175019 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-5919f375-f064-4013-9fdf-25fac9d60f9d admin admin] The token cache is being invalidate because project b7d504f39d52488abf8282bc0ec2786c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:39:02.175311 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-5919f375-f064-4013-9fdf-25fac9d60f9d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:39:02.175950 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-5919f375-f064-4013-9fdf-25fac9d60f9d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:39:02.177495 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1141/2307] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:39:02 2022] DELETE /identity/v3/projects/b7d504f39d52488abf8282bc0ec2786c => generated 0 bytes in 107 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:39:04.811374 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a01fff73-6e9f-4231-95fe-913ba84ba6dc None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:39:04.811887 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a01fff73-6e9f-4231-95fe-913ba84ba6dc None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:39:04.812072 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a01fff73-6e9f-4231-95fe-913ba84ba6dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:39:04.832328 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-a01fff73-6e9f-4231-95fe-913ba84ba6dc None None] MFA Rules not processed for user `050b3aab44bb4b17aa4a1ba7cccdb231`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:39:04.833576 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-a01fff73-6e9f-4231-95fe-913ba84ba6dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:39:04.896353 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1167/2308] 10.222.0.25 () {58 vars in 1009 bytes} [Sun Sep 25 04:39:04 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:39:04.909221 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5919f375-f064-4013-9fdf-25fac9d60f9d admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:39:04.910626 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-5919f375-f064-4013-9fdf-25fac9d60f9d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:39:04.981739 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5919f375-f064-4013-9fdf-25fac9d60f9d admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:39:04.987064 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13e3d917-f70c-4711-9c49-7a50b4fb0022 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '23ea88f46e7740d295148a93ade29dd9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:39:04.987622 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13e3d917-f70c-4711-9c49-7a50b4fb0022 service swift] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:39:04.987910 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13e3d917-f70c-4711-9c49-7a50b4fb0022 service swift] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:39:04.988188 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13e3d917-f70c-4711-9c49-7a50b4fb0022 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:39:04.997274 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13e3d917-f70c-4711-9c49-7a50b4fb0022 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:39:04.998879 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13e3d917-f70c-4711-9c49-7a50b4fb0022 service swift] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:39:05.007316 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1142/2309] 10.222.0.25 () {60 vars in 1384 bytes} [Sun Sep 25 04:39:04 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 25 04:39:14.889082 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=34 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:39:15.003733 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=53,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:39:44.058376 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a01fff73-6e9f-4231-95fe-913ba84ba6dc None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:39:44.059812 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-a01fff73-6e9f-4231-95fe-913ba84ba6dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:39:44.115849 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a01fff73-6e9f-4231-95fe-913ba84ba6dc None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:39:44.119438 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f3cf454-863e-47e7-9de7-ee7e6872e623 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:39:44.120042 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f3cf454-863e-47e7-9de7-ee7e6872e623 service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:39:44.120324 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f3cf454-863e-47e7-9de7-ee7e6872e623 service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:39:44.120593 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f3cf454-863e-47e7-9de7-ee7e6872e623 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:39:44.122322 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-4f3cf454-863e-47e7-9de7-ee7e6872e623 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:39:44.183084 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f3cf454-863e-47e7-9de7-ee7e6872e623 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:39:44.184435 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f3cf454-863e-47e7-9de7-ee7e6872e623 service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:39:44.194660 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1168/2310] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:39:44 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 137 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:39:54.190596 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=50 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:40:03.715563 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ddda85-0c0a-489c-af02-2f8a0b88d0ce None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:40:03.716089 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ddda85-0c0a-489c-af02-2f8a0b88d0ce None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:40:03.716185 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ddda85-0c0a-489c-af02-2f8a0b88d0ce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:40:03.736935 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-f7ddda85-0c0a-489c-af02-2f8a0b88d0ce None None] MFA Rules not processed for user `c50ba6f9a164463eae9a248936e7bf80`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:40:03.738133 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-f7ddda85-0c0a-489c-af02-2f8a0b88d0ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:40:03.792202 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1143/2311] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:40:03 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:40:13.785701 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=25 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:40:23.347713 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff6b1750-cd88-4b46-8235-4752c7ce7db3 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:40:23.348245 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff6b1750-cd88-4b46-8235-4752c7ce7db3 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:40:23.348354 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff6b1750-cd88-4b46-8235-4752c7ce7db3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:40:23.368828 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-ff6b1750-cd88-4b46-8235-4752c7ce7db3 None None] MFA Rules not processed for user `050b3aab44bb4b17aa4a1ba7cccdb231`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:40:23.370029 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-ff6b1750-cd88-4b46-8235-4752c7ce7db3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:40:23.377888 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1169/2312] 10.222.0.25 () {58 vars in 1009 bytes} [Sun Sep 25 04:40:23 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:40:23.418119 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7ddda85-0c0a-489c-af02-2f8a0b88d0ce None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:40:23.430041 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7ddda85-0c0a-489c-af02-2f8a0b88d0ce None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:40:23.435518 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b688e09d-a1fa-4c0e-8db3-db3fdba94fdc service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '23ea88f46e7740d295148a93ade29dd9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:40:23.436240 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b688e09d-a1fa-4c0e-8db3-db3fdba94fdc service swift] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:40:23.436519 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b688e09d-a1fa-4c0e-8db3-db3fdba94fdc service swift] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:40:23.436833 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b688e09d-a1fa-4c0e-8db3-db3fdba94fdc service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:40:23.444579 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b688e09d-a1fa-4c0e-8db3-db3fdba94fdc service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:40:23.446194 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b688e09d-a1fa-4c0e-8db3-db3fdba94fdc service swift] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:40:23.454775 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1144/2313] 10.222.0.25 () {60 vars in 1384 bytes} [Sun Sep 25 04:40:23 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 25 04:40:33.363023 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=3 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:40:33.451798 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=6 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:40:38.999707 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0bebf2d-e7f0-4010-a9c1-6a7c930280c6 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:40:39.000350 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0bebf2d-e7f0-4010-a9c1-6a7c930280c6 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:40:39.000350 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0bebf2d-e7f0-4010-a9c1-6a7c930280c6 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:40:39.001796 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1170/2314] 10.222.0.25 () {54 vars in 921 bytes} [Sun Sep 25 04:40:38 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:40:39.008117 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59417361-769c-497d-88b7-27ac867c0136 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:40:39.008552 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59417361-769c-497d-88b7-27ac867c0136 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:40:39.008840 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59417361-769c-497d-88b7-27ac867c0136 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:40:39.029135 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-59417361-769c-497d-88b7-27ac867c0136 None None] MFA Rules not processed for user `e703b42c53974daaa608b0ccc3f018b1`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:40:39.030480 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-59417361-769c-497d-88b7-27ac867c0136 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:40:39.101483 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1145/2315] 10.222.0.25 () {58 vars in 1018 bytes} [Sun Sep 25 04:40:39 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 04:40:39.115203 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0bebf2d-e7f0-4010-a9c1-6a7c930280c6 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:40:39.116869 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-b0bebf2d-e7f0-4010-a9c1-6a7c930280c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:40:39.178739 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0bebf2d-e7f0-4010-a9c1-6a7c930280c6 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:40:39.182269 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c69457cf-de11-419b-9f1c-8604426c4003 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '316eaa67d5464091af47efd489b96b87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:40:39.182866 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c69457cf-de11-419b-9f1c-8604426c4003 service glance] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:40:39.183142 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c69457cf-de11-419b-9f1c-8604426c4003 service glance] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:40:39.183411 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c69457cf-de11-419b-9f1c-8604426c4003 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:40:39.194034 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c69457cf-de11-419b-9f1c-8604426c4003 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:40:39.195436 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c69457cf-de11-419b-9f1c-8604426c4003 service glance] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:40:39.207047 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1171/2316] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:40:39 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:40:39.275950 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b232c1e-5ceb-49ab-8b12-75683ec5b618 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:40:39.276187 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b232c1e-5ceb-49ab-8b12-75683ec5b618 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:40:39.276469 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b232c1e-5ceb-49ab-8b12-75683ec5b618 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:40:39.277842 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1146/2317] 10.222.0.25 () {54 vars in 921 bytes} [Sun Sep 25 04:40:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:40:39.282881 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a4348e-047c-4ed7-bb7d-3fa359f0f254 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:40:39.283172 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a4348e-047c-4ed7-bb7d-3fa359f0f254 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:40:39.283452 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a4348e-047c-4ed7-bb7d-3fa359f0f254 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:40:39.303818 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-c9a4348e-047c-4ed7-bb7d-3fa359f0f254 None None] MFA Rules not processed for user `0268e88e0c5645a98bab5664be6e49bc`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:40:39.305089 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-c9a4348e-047c-4ed7-bb7d-3fa359f0f254 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:40:39.312415 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1172/2318] 10.222.0.25 () {58 vars in 1018 bytes} [Sun Sep 25 04:40:39 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:40:39.325489 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b232c1e-5ceb-49ab-8b12-75683ec5b618 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:40:39.327225 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-1b232c1e-5ceb-49ab-8b12-75683ec5b618 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:40:39.340472 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b232c1e-5ceb-49ab-8b12-75683ec5b618 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:40:39.344420 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e16ef55-e316-4dee-a716-99e3ff1f5db7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:40:39.345059 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e16ef55-e316-4dee-a716-99e3ff1f5db7 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:40:39.345328 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e16ef55-e316-4dee-a716-99e3ff1f5db7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:40:39.345598 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e16ef55-e316-4dee-a716-99e3ff1f5db7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:40:39.352913 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e16ef55-e316-4dee-a716-99e3ff1f5db7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:40:39.354340 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e16ef55-e316-4dee-a716-99e3ff1f5db7 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:40:39.364937 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1147/2319] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:40:39 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:40:39.463839 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d5a3345-1b93-40eb-b099-cb98b06a2d00 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:40:39.464205 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d5a3345-1b93-40eb-b099-cb98b06a2d00 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:40:39.464600 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d5a3345-1b93-40eb-b099-cb98b06a2d00 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:40:39.466879 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1173/2320] 10.222.0.25 () {54 vars in 921 bytes} [Sun Sep 25 04:40:39 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:40:39.473327 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70307fd8-30b1-42b1-8208-ac5b76a684ff None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:40:39.473621 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70307fd8-30b1-42b1-8208-ac5b76a684ff None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:40:39.473898 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70307fd8-30b1-42b1-8208-ac5b76a684ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:40:39.494366 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-70307fd8-30b1-42b1-8208-ac5b76a684ff None None] MFA Rules not processed for user `e703b42c53974daaa608b0ccc3f018b1`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:40:39.495585 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-70307fd8-30b1-42b1-8208-ac5b76a684ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:40:39.505715 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1148/2321] 10.222.0.25 () {58 vars in 1018 bytes} [Sun Sep 25 04:40:39 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:40:39.516338 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d5a3345-1b93-40eb-b099-cb98b06a2d00 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:40:39.527323 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d5a3345-1b93-40eb-b099-cb98b06a2d00 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:40:39.531238 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-764ee326-a881-4c15-adfa-e588373cd237 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:40:39.531859 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-764ee326-a881-4c15-adfa-e588373cd237 service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:40:39.532162 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-764ee326-a881-4c15-adfa-e588373cd237 service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:40:39.532435 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-764ee326-a881-4c15-adfa-e588373cd237 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:40:39.542471 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-764ee326-a881-4c15-adfa-e588373cd237 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:40:39.543884 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-764ee326-a881-4c15-adfa-e588373cd237 service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:40:39.554987 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1174/2322] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:40:39 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:40:47.805480 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2873ea13-04a8-4b17-9001-5d0ac4dc17f6 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:40:47.805985 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2873ea13-04a8-4b17-9001-5d0ac4dc17f6 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:40:47.806070 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2873ea13-04a8-4b17-9001-5d0ac4dc17f6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:40:47.825793 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-2873ea13-04a8-4b17-9001-5d0ac4dc17f6 None None] MFA Rules not processed for user `050b3aab44bb4b17aa4a1ba7cccdb231`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:40:47.827571 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-2873ea13-04a8-4b17-9001-5d0ac4dc17f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:40:47.839066 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1149/2323] 10.222.0.25 () {58 vars in 1009 bytes} [Sun Sep 25 04:40:47 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:40:47.851539 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-764ee326-a881-4c15-adfa-e588373cd237 service neutron] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:40:47.863591 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-764ee326-a881-4c15-adfa-e588373cd237 service neutron] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:40:47.868611 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edb9d756-0b94-40e7-9009-786081842f85 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '23ea88f46e7740d295148a93ade29dd9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:40:47.869257 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edb9d756-0b94-40e7-9009-786081842f85 service swift] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:40:47.869590 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edb9d756-0b94-40e7-9009-786081842f85 service swift] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:40:47.869868 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edb9d756-0b94-40e7-9009-786081842f85 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:40:47.878719 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edb9d756-0b94-40e7-9009-786081842f85 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:40:47.880291 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edb9d756-0b94-40e7-9009-786081842f85 service swift] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:40:47.888228 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1175/2324] 10.222.0.25 () {60 vars in 1384 bytes} [Sun Sep 25 04:40:47 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 25 04:40:54.900000 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2873ea13-04a8-4b17-9001-5d0ac4dc17f6 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:40:54.910518 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2873ea13-04a8-4b17-9001-5d0ac4dc17f6 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:40:54.914150 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-febb8fd9-571a-40a9-bb43-28a0dafa0dd4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:40:54.914754 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-febb8fd9-571a-40a9-bb43-28a0dafa0dd4 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:40:54.915031 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-febb8fd9-571a-40a9-bb43-28a0dafa0dd4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:40:54.915297 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-febb8fd9-571a-40a9-bb43-28a0dafa0dd4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:40:54.917234 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-febb8fd9-571a-40a9-bb43-28a0dafa0dd4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:40:54.989794 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-febb8fd9-571a-40a9-bb43-28a0dafa0dd4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:40:54.991507 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-febb8fd9-571a-40a9-bb43-28a0dafa0dd4 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:40:55.002284 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1150/2325] 10.222.0.25 () {60 vars in 1396 bytes} [Sun Sep 25 04:40:54 2022] GET /identity/v3/auth/tokens?allow_expired=1 => generated 3270 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:40:56.145005 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edb9d756-0b94-40e7-9009-786081842f85 service swift] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:40:56.155905 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edb9d756-0b94-40e7-9009-786081842f85 service swift] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:40:56.159779 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93d1d890-ecaf-4f7f-a8c7-6c1b9471303a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:40:56.160393 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d1d890-ecaf-4f7f-a8c7-6c1b9471303a service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:40:56.160827 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d1d890-ecaf-4f7f-a8c7-6c1b9471303a service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:40:56.161007 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d1d890-ecaf-4f7f-a8c7-6c1b9471303a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:40:56.162948 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-93d1d890-ecaf-4f7f-a8c7-6c1b9471303a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:40:56.177396 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93d1d890-ecaf-4f7f-a8c7-6c1b9471303a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:40:56.178839 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93d1d890-ecaf-4f7f-a8c7-6c1b9471303a service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:40:56.190514 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1176/2326] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:40:56 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:40:56.933969 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-febb8fd9-571a-40a9-bb43-28a0dafa0dd4 service neutron] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:40:56.943483 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-febb8fd9-571a-40a9-bb43-28a0dafa0dd4 service neutron] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:40:56.947240 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-588f0d0e-5240-47e9-a67b-a6dd08005560 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:40:56.947853 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-588f0d0e-5240-47e9-a67b-a6dd08005560 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:40:56.948143 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-588f0d0e-5240-47e9-a67b-a6dd08005560 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:40:56.948416 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-588f0d0e-5240-47e9-a67b-a6dd08005560 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:40:56.950261 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-588f0d0e-5240-47e9-a67b-a6dd08005560 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:40:56.967907 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-588f0d0e-5240-47e9-a67b-a6dd08005560 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:40:56.968156 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-588f0d0e-5240-47e9-a67b-a6dd08005560 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:40:56.980770 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1151/2327] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:40:56 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 47 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:41:02.948036 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93d1d890-ecaf-4f7f-a8c7-6c1b9471303a service neutron] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:41:02.957371 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93d1d890-ecaf-4f7f-a8c7-6c1b9471303a service neutron] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:41:02.961424 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fcb7bb5-5034-4f97-bcd3-77a355082bd0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:41:02.962158 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fcb7bb5-5034-4f97-bcd3-77a355082bd0 service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:41:02.962493 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fcb7bb5-5034-4f97-bcd3-77a355082bd0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:41:02.962767 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fcb7bb5-5034-4f97-bcd3-77a355082bd0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:41:02.964781 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-2fcb7bb5-5034-4f97-bcd3-77a355082bd0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:41:03.028623 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fcb7bb5-5034-4f97-bcd3-77a355082bd0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:41:03.030092 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fcb7bb5-5034-4f97-bcd3-77a355082bd0 service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:41:03.039959 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1177/2328] 10.222.0.25 () {60 vars in 1396 bytes} [Sun Sep 25 04:41:02 2022] GET /identity/v3/auth/tokens?allow_expired=1 => generated 3266 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:41:06.975780 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=77 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:41:10.447585 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ae6c9d0-82a1-4670-b578-dc96460dabbe None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:41:10.448207 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ae6c9d0-82a1-4670-b578-dc96460dabbe None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:41:10.448207 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ae6c9d0-82a1-4670-b578-dc96460dabbe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:41:10.467839 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-8ae6c9d0-82a1-4670-b578-dc96460dabbe None None] MFA Rules not processed for user `2666d4774776498bbdbce539296e76a9`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:41:10.469089 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-8ae6c9d0-82a1-4670-b578-dc96460dabbe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:41:10.476171 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1152/2329] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:41:10 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:41:12.953191 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fcb7bb5-5034-4f97-bcd3-77a355082bd0 service neutron] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:41:12.957551 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=76 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:41:12.963703 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fcb7bb5-5034-4f97-bcd3-77a355082bd0 service neutron] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:41:12.967403 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91c4e17d-518e-42fa-b932-52ae6e9df187 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:41:12.968009 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c4e17d-518e-42fa-b932-52ae6e9df187 service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:41:12.968292 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c4e17d-518e-42fa-b932-52ae6e9df187 service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:41:12.968564 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c4e17d-518e-42fa-b932-52ae6e9df187 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:41:12.970363 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-91c4e17d-518e-42fa-b932-52ae6e9df187 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:41:12.979868 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91c4e17d-518e-42fa-b932-52ae6e9df187 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:41:12.981391 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91c4e17d-518e-42fa-b932-52ae6e9df187 service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:41:12.990038 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1178/2330] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:41:12 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:41:19.358058 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ae6c9d0-82a1-4670-b578-dc96460dabbe None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:41:19.359611 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-8ae6c9d0-82a1-4670-b578-dc96460dabbe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:41:19.379261 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ae6c9d0-82a1-4670-b578-dc96460dabbe None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:41:19.387838 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc869d50-bc0a-43e0-88ec-f0e3d12a87e0 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e703b42c53974daaa608b0ccc3f018b1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'ResellerAdmin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:41:19.388545 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc869d50-bc0a-43e0-88ec-f0e3d12a87e0 service nova] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:41:19.388866 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc869d50-bc0a-43e0-88ec-f0e3d12a87e0 service nova] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:41:19.389142 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc869d50-bc0a-43e0-88ec-f0e3d12a87e0 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:41:19.391315 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-bc869d50-bc0a-43e0-88ec-f0e3d12a87e0 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:41:19.405634 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc869d50-bc0a-43e0-88ec-f0e3d12a87e0 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:41:19.407228 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc869d50-bc0a-43e0-88ec-f0e3d12a87e0 service nova] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:41:19.420603 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1153/2331] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:41:19 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 63 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:41:21.789613 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dc5214a-059e-4edb-872a-7c32cd4c1ee2 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:41:21.790173 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dc5214a-059e-4edb-872a-7c32cd4c1ee2 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:41:21.790319 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dc5214a-059e-4edb-872a-7c32cd4c1ee2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:41:21.811518 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-8dc5214a-059e-4edb-872a-7c32cd4c1ee2 None None] MFA Rules not processed for user `050b3aab44bb4b17aa4a1ba7cccdb231`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:41:21.812887 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-8dc5214a-059e-4edb-872a-7c32cd4c1ee2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:41:21.821321 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1179/2332] 10.222.0.25 () {58 vars in 967 bytes} [Sun Sep 25 04:41:21 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:41:21.834748 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc869d50-bc0a-43e0-88ec-f0e3d12a87e0 service nova] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:41:21.846744 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc869d50-bc0a-43e0-88ec-f0e3d12a87e0 service nova] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:41:21.852303 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-454c39cf-d6c0-4cc8-8048-77cd69f57637 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '23ea88f46e7740d295148a93ade29dd9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:41:21.853060 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-454c39cf-d6c0-4cc8-8048-77cd69f57637 service swift] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:41:21.853418 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-454c39cf-d6c0-4cc8-8048-77cd69f57637 service swift] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:41:21.853824 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-454c39cf-d6c0-4cc8-8048-77cd69f57637 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:41:21.862557 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-454c39cf-d6c0-4cc8-8048-77cd69f57637 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:41:21.864807 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-454c39cf-d6c0-4cc8-8048-77cd69f57637 service swift] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:41:21.873120 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1154/2333] 10.222.0.25 () {60 vars in 1384 bytes} [Sun Sep 25 04:41:21 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 25 04:41:23.616149 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dc5214a-059e-4edb-872a-7c32cd4c1ee2 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:41:23.629227 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dc5214a-059e-4edb-872a-7c32cd4c1ee2 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:41:23.635476 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-290a3584-0c3c-4fbe-ae89-39d6c49ff428 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e703b42c53974daaa608b0ccc3f018b1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'ResellerAdmin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:41:23.636093 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-290a3584-0c3c-4fbe-ae89-39d6c49ff428 service nova] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:41:23.636373 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-290a3584-0c3c-4fbe-ae89-39d6c49ff428 service nova] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:41:23.636655 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-290a3584-0c3c-4fbe-ae89-39d6c49ff428 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:41:23.638460 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-290a3584-0c3c-4fbe-ae89-39d6c49ff428 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:41:23.702484 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-290a3584-0c3c-4fbe-ae89-39d6c49ff428 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:41:23.704168 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-290a3584-0c3c-4fbe-ae89-39d6c49ff428 service nova] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:41:23.715115 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1180/2334] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:41:23 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:41:27.507076 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc16c358-f20d-4e77-b175-3905635b9d5f None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:41:27.507582 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc16c358-f20d-4e77-b175-3905635b9d5f None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:41:27.507689 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc16c358-f20d-4e77-b175-3905635b9d5f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:41:27.528448 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-cc16c358-f20d-4e77-b175-3905635b9d5f None None] MFA Rules not processed for user `329e3f9810b648218dd45a8b489adb1d`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:41:27.529826 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-cc16c358-f20d-4e77-b175-3905635b9d5f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:41:27.593513 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1155/2335] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:41:27 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:41:27.958891 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-290a3584-0c3c-4fbe-ae89-39d6c49ff428 service nova] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:41:27.968670 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-290a3584-0c3c-4fbe-ae89-39d6c49ff428 service nova] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:41:27.972522 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b1eacac-d6d2-4aa2-bba0-7b9536fe3294 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:41:27.973167 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b1eacac-d6d2-4aa2-bba0-7b9536fe3294 service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:41:27.973460 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b1eacac-d6d2-4aa2-bba0-7b9536fe3294 service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:41:27.973788 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b1eacac-d6d2-4aa2-bba0-7b9536fe3294 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:41:27.975658 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-9b1eacac-d6d2-4aa2-bba0-7b9536fe3294 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:41:27.985446 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b1eacac-d6d2-4aa2-bba0-7b9536fe3294 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:41:27.986854 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b1eacac-d6d2-4aa2-bba0-7b9536fe3294 service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:41:27.995803 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1181/2336] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:41:27 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:41:28.431431 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc16c358-f20d-4e77-b175-3905635b9d5f None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:41:28.432978 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-cc16c358-f20d-4e77-b175-3905635b9d5f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:41:28.509678 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc16c358-f20d-4e77-b175-3905635b9d5f None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:41:28.514758 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-086b796f-e3ec-4a21-8a2a-560616f66b78 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '676d04ef5a984edcaad63d9156568660', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:41:28.515371 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-086b796f-e3ec-4a21-8a2a-560616f66b78 service placement] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:41:28.515653 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-086b796f-e3ec-4a21-8a2a-560616f66b78 service placement] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:41:28.515927 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-086b796f-e3ec-4a21-8a2a-560616f66b78 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:41:28.517666 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-086b796f-e3ec-4a21-8a2a-560616f66b78 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:41:28.529011 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-086b796f-e3ec-4a21-8a2a-560616f66b78 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:41:28.530606 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-086b796f-e3ec-4a21-8a2a-560616f66b78 service placement] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:41:28.540945 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1156/2337] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:41:28 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 110 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:41:37.992381 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=42 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:41:38.536801 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=72 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:41:44.712357 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b1eacac-d6d2-4aa2-bba0-7b9536fe3294 service neutron] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:41:44.713946 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-9b1eacac-d6d2-4aa2-bba0-7b9536fe3294 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:41:44.731512 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b1eacac-d6d2-4aa2-bba0-7b9536fe3294 service neutron] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:41:44.737036 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-992f392f-eb9c-468a-8cc4-ced23e27bb79 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e703b42c53974daaa608b0ccc3f018b1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'ResellerAdmin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:41:44.737642 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-992f392f-eb9c-468a-8cc4-ced23e27bb79 service nova] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:41:44.737923 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-992f392f-eb9c-468a-8cc4-ced23e27bb79 service nova] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:41:44.738347 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-992f392f-eb9c-468a-8cc4-ced23e27bb79 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:41:44.740072 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-992f392f-eb9c-468a-8cc4-ced23e27bb79 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:41:44.753690 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-992f392f-eb9c-468a-8cc4-ced23e27bb79 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:41:44.755524 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-992f392f-eb9c-468a-8cc4-ced23e27bb79 service nova] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:41:44.767004 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1182/2338] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:41:44 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 55 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:41:54.762306 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=6 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:42:16.931690 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-086b796f-e3ec-4a21-8a2a-560616f66b78 service placement] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:42:16.944769 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-086b796f-e3ec-4a21-8a2a-560616f66b78 service placement] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:42:16.950288 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9af658d-65f6-4fc9-b528-9d8b2557fafc service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e703b42c53974daaa608b0ccc3f018b1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'ResellerAdmin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:42:16.950914 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9af658d-65f6-4fc9-b528-9d8b2557fafc service nova] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:42:16.951199 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9af658d-65f6-4fc9-b528-9d8b2557fafc service nova] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:42:16.951477 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9af658d-65f6-4fc9-b528-9d8b2557fafc service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:42:16.953241 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-a9af658d-65f6-4fc9-b528-9d8b2557fafc service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:42:16.964874 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9af658d-65f6-4fc9-b528-9d8b2557fafc service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:42:16.966462 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9af658d-65f6-4fc9-b528-9d8b2557fafc service nova] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:42:16.975800 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1157/2339] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:42:16 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:42:26.971332 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=6 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:42:33.111435 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dbadffc-ac7c-4391-824f-64af723c7773 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:42:33.111955 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dbadffc-ac7c-4391-824f-64af723c7773 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:42:33.111955 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dbadffc-ac7c-4391-824f-64af723c7773 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:42:33.131850 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-6dbadffc-ac7c-4391-824f-64af723c7773 None None] MFA Rules not processed for user `c50ba6f9a164463eae9a248936e7bf80`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:42:33.133086 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-6dbadffc-ac7c-4391-824f-64af723c7773 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:42:33.139921 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1183/2340] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:42:33 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:42:43.125818 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=3 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:42:50.552520 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9af658d-65f6-4fc9-b528-9d8b2557fafc service nova] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:42:50.565091 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9af658d-65f6-4fc9-b528-9d8b2557fafc service nova] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:42:50.572206 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a47d95f-429f-4366-87ca-82b59b89f7ae service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e703b42c53974daaa608b0ccc3f018b1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'ResellerAdmin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:42:50.572856 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a47d95f-429f-4366-87ca-82b59b89f7ae service nova] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:42:50.573271 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a47d95f-429f-4366-87ca-82b59b89f7ae service nova] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:42:50.573553 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a47d95f-429f-4366-87ca-82b59b89f7ae service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:42:50.575464 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-9a47d95f-429f-4366-87ca-82b59b89f7ae service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:42:50.601638 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a47d95f-429f-4366-87ca-82b59b89f7ae service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:42:50.603380 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a47d95f-429f-4366-87ca-82b59b89f7ae service nova] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:42:50.615484 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1158/2341] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:42:50 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 64 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:42:56.369005 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7de09d08-3e24-473c-b3e5-d73c94641bf2 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:42:56.369549 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7de09d08-3e24-473c-b3e5-d73c94641bf2 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:42:56.369549 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7de09d08-3e24-473c-b3e5-d73c94641bf2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:42:56.389250 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-7de09d08-3e24-473c-b3e5-d73c94641bf2 None None] MFA Rules not processed for user `e7354aa144df4ad69cd7d12f11b1d8a5`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:42:56.392614 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-7de09d08-3e24-473c-b3e5-d73c94641bf2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:42:56.400829 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1184/2342] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:42:56 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:43:00.609760 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=6 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:43:05.590701 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a47d95f-429f-4366-87ca-82b59b89f7ae service nova] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:05.600759 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a47d95f-429f-4366-87ca-82b59b89f7ae service nova] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:05.604581 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4544cd46-5a51-46c9-9f7c-5e8155616d28 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:05.605516 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4544cd46-5a51-46c9-9f7c-5e8155616d28 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:05.605688 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4544cd46-5a51-46c9-9f7c-5e8155616d28 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:05.605959 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4544cd46-5a51-46c9-9f7c-5e8155616d28 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:05.607863 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-4544cd46-5a51-46c9-9f7c-5e8155616d28 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:43:05.674745 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4544cd46-5a51-46c9-9f7c-5e8155616d28 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:05.676696 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4544cd46-5a51-46c9-9f7c-5e8155616d28 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:05.686637 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1159/2343] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:43:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:43:06.383629 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=3 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:43:09.476387 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7de09d08-3e24-473c-b3e5-d73c94641bf2 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:09.485932 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-7de09d08-3e24-473c-b3e5-d73c94641bf2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:43:09.491637 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4544cd46-5a51-46c9-9f7c-5e8155616d28 service neutron] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:09.493151 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-4544cd46-5a51-46c9-9f7c-5e8155616d28 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:43:09.520094 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7de09d08-3e24-473c-b3e5-d73c94641bf2 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:09.527359 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4544cd46-5a51-46c9-9f7c-5e8155616d28 service neutron] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:09.528901 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5268596d-2ced-48bc-a945-e92f9d008fc7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:09.529993 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5268596d-2ced-48bc-a945-e92f9d008fc7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:09.531228 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5268596d-2ced-48bc-a945-e92f9d008fc7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:09.531658 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5268596d-2ced-48bc-a945-e92f9d008fc7 admin admin] PATH_INFO: `/v3/users/2666d4774776498bbdbce539296e76a9` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:09.537099 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5268596d-2ced-48bc-a945-e92f9d008fc7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2666d4774776498bbdbce539296e76a9)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:09.539821 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5268596d-2ced-48bc-a945-e92f9d008fc7 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:09.560692 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9eace5e-bd19-44fb-aa3f-b022219ca191 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '676d04ef5a984edcaad63d9156568660', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:09.561839 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9eace5e-bd19-44fb-aa3f-b022219ca191 service placement] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:09.563898 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9eace5e-bd19-44fb-aa3f-b022219ca191 service placement] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:09.564340 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9eace5e-bd19-44fb-aa3f-b022219ca191 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:09.575300 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-c9eace5e-bd19-44fb-aa3f-b022219ca191 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:43:09.610471 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-5268596d-2ced-48bc-a945-e92f9d008fc7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2666d4774776498bbdbce539296e76a9'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:09.624251 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-5268596d-2ced-48bc-a945-e92f9d008fc7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2666d4774776498bbdbce539296e76a9'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:09.630930 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9eace5e-bd19-44fb-aa3f-b022219ca191 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:09.632552 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9eace5e-bd19-44fb-aa3f-b022219ca191 service placement] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:09.656390 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-5268596d-2ced-48bc-a945-e92f9d008fc7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2666d4774776498bbdbce539296e76a9'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:09.658364 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-5268596d-2ced-48bc-a945-e92f9d008fc7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2666d4774776498bbdbce539296e76a9'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:09.666508 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1160/2344] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:43:09 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 175 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:43:09.681945 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-5268596d-2ced-48bc-a945-e92f9d008fc7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2666d4774776498bbdbce539296e76a9'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:09.692310 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1185/2345] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:43:09 2022] DELETE /identity/v3/users/2666d4774776498bbdbce539296e76a9 => generated 0 bytes in 216 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:43:10.040263 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9eace5e-bd19-44fb-aa3f-b022219ca191 service placement] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:10.053098 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9eace5e-bd19-44fb-aa3f-b022219ca191 service placement] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:10.120968 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51c56cc2-ddb1-4f63-802d-7a778b3a185e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:10.121620 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51c56cc2-ddb1-4f63-802d-7a778b3a185e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:10.121904 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51c56cc2-ddb1-4f63-802d-7a778b3a185e admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:10.122173 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51c56cc2-ddb1-4f63-802d-7a778b3a185e admin admin] PATH_INFO: `/v3/projects/f6f23cf0323f48069e21acf059cbd1f0` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:10.123490 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51c56cc2-ddb1-4f63-802d-7a778b3a185e admin admin] RBAC: Authorizing `identity:delete_project(project_id=f6f23cf0323f48069e21acf059cbd1f0)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:10.125648 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51c56cc2-ddb1-4f63-802d-7a778b3a185e admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:10.203851 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-51c56cc2-ddb1-4f63-802d-7a778b3a185e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f6f23cf0323f48069e21acf059cbd1f0', 'request_id': 'req-51c56cc2-ddb1-4f63-802d-7a778b3a185e'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:10.215417 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-51c56cc2-ddb1-4f63-802d-7a778b3a185e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f6f23cf0323f48069e21acf059cbd1f0', 'request_id': 'req-51c56cc2-ddb1-4f63-802d-7a778b3a185e'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:10.222937 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-51c56cc2-ddb1-4f63-802d-7a778b3a185e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f6f23cf0323f48069e21acf059cbd1f0', 'request_id': 'req-51c56cc2-ddb1-4f63-802d-7a778b3a185e'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:10.237844 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-51c56cc2-ddb1-4f63-802d-7a778b3a185e admin admin] The token cache is being invalidate because project f6f23cf0323f48069e21acf059cbd1f0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:43:10.238622 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-51c56cc2-ddb1-4f63-802d-7a778b3a185e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:10.239529 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-51c56cc2-ddb1-4f63-802d-7a778b3a185e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:10.241635 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1161/2346] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:43:10 2022] DELETE /identity/v3/projects/f6f23cf0323f48069e21acf059cbd1f0 => generated 0 bytes in 202 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:43:10.246560 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1536127a-8421-447a-a5b4-e525c08ae979 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:10.247061 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1536127a-8421-447a-a5b4-e525c08ae979 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:10.247332 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1536127a-8421-447a-a5b4-e525c08ae979 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:10.290220 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-1536127a-8421-447a-a5b4-e525c08ae979 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:43:10.291758 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-1536127a-8421-447a-a5b4-e525c08ae979 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:43:10.387576 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1186/2347] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:43:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 142 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:43:10.396207 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db80a58b-c49e-428d-bdcd-41d12888744f None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:10.396498 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db80a58b-c49e-428d-bdcd-41d12888744f None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:10.396827 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db80a58b-c49e-428d-bdcd-41d12888744f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:10.419996 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-db80a58b-c49e-428d-bdcd-41d12888744f None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:43:10.421390 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-db80a58b-c49e-428d-bdcd-41d12888744f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:43:10.430255 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1162/2348] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:43:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:43:10.434719 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1536127a-8421-447a-a5b4-e525c08ae979 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:10.446985 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1536127a-8421-447a-a5b4-e525c08ae979 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:10.452836 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-861953e6-742e-4f45-a958-e7ed0c3aea04 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:10.453456 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-861953e6-742e-4f45-a958-e7ed0c3aea04 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:10.453737 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-861953e6-742e-4f45-a958-e7ed0c3aea04 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:10.454208 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-861953e6-742e-4f45-a958-e7ed0c3aea04 admin admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:10.454864 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-861953e6-742e-4f45-a958-e7ed0c3aea04 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:43:10.455172 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-861953e6-742e-4f45-a958-e7ed0c3aea04 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:10.458232 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-861953e6-742e-4f45-a958-e7ed0c3aea04 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:10.478021 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1187/2349] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Sep 25 04:43:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:43:10.478779 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db80a58b-c49e-428d-bdcd-41d12888744f None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:10.495192 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:43:10.505914 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db80a58b-c49e-428d-bdcd-41d12888744f None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:10.521620 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-423146b4-22e5-476e-9aed-37eeccf1ea47 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:10.522550 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-423146b4-22e5-476e-9aed-37eeccf1ea47 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:10.522824 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-423146b4-22e5-476e-9aed-37eeccf1ea47 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:10.523091 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-423146b4-22e5-476e-9aed-37eeccf1ea47 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:10.523728 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-423146b4-22e5-476e-9aed-37eeccf1ea47 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:10.525750 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-423146b4-22e5-476e-9aed-37eeccf1ea47 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:10.577231 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1163/2350] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:43:10 2022] POST /identity/v3/projects => generated 371 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:43:10.585364 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-861953e6-742e-4f45-a958-e7ed0c3aea04 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:10.603411 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-861953e6-742e-4f45-a958-e7ed0c3aea04 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:10.778104 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-168218e0-8a5d-4f12-9276-ae50f683f671 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:10.779824 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-168218e0-8a5d-4f12-9276-ae50f683f671 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:10.781266 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-168218e0-8a5d-4f12-9276-ae50f683f671 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:10.783569 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-168218e0-8a5d-4f12-9276-ae50f683f671 admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:10.787317 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-168218e0-8a5d-4f12-9276-ae50f683f671 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:10.792059 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-168218e0-8a5d-4f12-9276-ae50f683f671 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:10.838404 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:43:10.871289 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1188/2351] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:43:10 2022] POST /identity/v3/users => generated 286 bytes in 289 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:43:10.875103 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-423146b4-22e5-476e-9aed-37eeccf1ea47 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:10.892200 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-423146b4-22e5-476e-9aed-37eeccf1ea47 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:10.906861 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9148e679-a1a3-4d60-b5ff-0b7d49f8e92f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:10.907625 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9148e679-a1a3-4d60-b5ff-0b7d49f8e92f admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:10.907921 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9148e679-a1a3-4d60-b5ff-0b7d49f8e92f admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:10.908223 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9148e679-a1a3-4d60-b5ff-0b7d49f8e92f admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:10.909061 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9148e679-a1a3-4d60-b5ff-0b7d49f8e92f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:10.911628 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9148e679-a1a3-4d60-b5ff-0b7d49f8e92f admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:10.923851 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1164/2352] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:43:10 2022] GET /identity/v3/roles => generated 1420 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:43:10.935979 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-168218e0-8a5d-4f12-9276-ae50f683f671 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:10.949013 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-168218e0-8a5d-4f12-9276-ae50f683f671 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:10.957236 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f8e7c1a-7dcf-4898-b301-6128ff95d29c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:10.958121 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f8e7c1a-7dcf-4898-b301-6128ff95d29c admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:10.958456 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f8e7c1a-7dcf-4898-b301-6128ff95d29c admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:10.958830 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f8e7c1a-7dcf-4898-b301-6128ff95d29c admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:10.959724 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f8e7c1a-7dcf-4898-b301-6128ff95d29c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:10.962100 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f8e7c1a-7dcf-4898-b301-6128ff95d29c admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:10.981509 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1189/2353] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:43:10 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:43:10.982476 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9148e679-a1a3-4d60-b5ff-0b7d49f8e92f admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:11.002883 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9148e679-a1a3-4d60-b5ff-0b7d49f8e92f admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:11.008582 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8ecdb23-f8ab-4ec0-9f78-52a133ff759b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:11.009271 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8ecdb23-f8ab-4ec0-9f78-52a133ff759b admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:11.009723 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8ecdb23-f8ab-4ec0-9f78-52a133ff759b admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:11.009998 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8ecdb23-f8ab-4ec0-9f78-52a133ff759b admin admin] PATH_INFO: `/v3/projects/d3f8a3c6dad64c04a37c1f1abe51d6b8/users/a968a29f3730486aa8ebb9e1fd182470/roles/8f6cc574056c4278bc9778f14dc4b411` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:11.044233 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8ecdb23-f8ab-4ec0-9f78-52a133ff759b admin admin] RBAC: Authorizing `identity:create_grant(project_id=d3f8a3c6dad64c04a37c1f1abe51d6b8, user_id=a968a29f3730486aa8ebb9e1fd182470, role_id=8f6cc574056c4278bc9778f14dc4b411)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:11.050329 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8ecdb23-f8ab-4ec0-9f78-52a133ff759b admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:11.091284 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e887cdd9-f99c-4dda-b08d-953abd0284a3 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:11.091608 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e887cdd9-f99c-4dda-b08d-953abd0284a3 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:11.091883 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e887cdd9-f99c-4dda-b08d-953abd0284a3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:11.099744 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1165/2354] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:43:10 2022] PUT /identity/v3/projects/d3f8a3c6dad64c04a37c1f1abe51d6b8/users/a968a29f3730486aa8ebb9e1fd182470/roles/8f6cc574056c4278bc9778f14dc4b411 => generated 0 bytes in 97 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:43:11.130694 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-e887cdd9-f99c-4dda-b08d-953abd0284a3 None None] MFA Rules not processed for user `a968a29f3730486aa8ebb9e1fd182470`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:43:11.132869 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-e887cdd9-f99c-4dda-b08d-953abd0284a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:43:11.222462 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1190/2355] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:43:11 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:43:11.229294 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8ecdb23-f8ab-4ec0-9f78-52a133ff759b admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:11.231014 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-b8ecdb23-f8ab-4ec0-9f78-52a133ff759b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:43:11.311473 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8ecdb23-f8ab-4ec0-9f78-52a133ff759b admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:11.318155 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ba30f8a-7f39-4e0d-afc9-be6a22f0d25b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e703b42c53974daaa608b0ccc3f018b1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'ResellerAdmin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:11.318780 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ba30f8a-7f39-4e0d-afc9-be6a22f0d25b service nova] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:11.319064 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ba30f8a-7f39-4e0d-afc9-be6a22f0d25b service nova] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:11.319348 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ba30f8a-7f39-4e0d-afc9-be6a22f0d25b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:11.328441 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ba30f8a-7f39-4e0d-afc9-be6a22f0d25b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:11.330269 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ba30f8a-7f39-4e0d-afc9-be6a22f0d25b service nova] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:11.341754 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1166/2356] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:43:11 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 113 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:43:12.468102 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f727d66-aa5d-4f3f-a869-5e8c5dedab41 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:12.468592 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f727d66-aa5d-4f3f-a869-5e8c5dedab41 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:12.468737 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f727d66-aa5d-4f3f-a869-5e8c5dedab41 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:12.489194 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-9f727d66-aa5d-4f3f-a869-5e8c5dedab41 None None] MFA Rules not processed for user `16281c79f73e4063b52066966c393148`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:43:12.490394 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-9f727d66-aa5d-4f3f-a869-5e8c5dedab41 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:43:12.544828 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1191/2357] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:43:12 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:43:12.859311 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ba30f8a-7f39-4e0d-afc9-be6a22f0d25b service nova] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:12.860812 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-8ba30f8a-7f39-4e0d-afc9-be6a22f0d25b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:43:12.919943 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ba30f8a-7f39-4e0d-afc9-be6a22f0d25b service nova] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:12.924066 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d673703-67b1-43c3-bf19-93d65696ef42 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:12.924703 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d673703-67b1-43c3-bf19-93d65696ef42 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:12.924983 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d673703-67b1-43c3-bf19-93d65696ef42 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:12.925253 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d673703-67b1-43c3-bf19-93d65696ef42 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:12.927386 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-8d673703-67b1-43c3-bf19-93d65696ef42 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:43:12.937335 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d673703-67b1-43c3-bf19-93d65696ef42 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:12.938750 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d673703-67b1-43c3-bf19-93d65696ef42 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:12.948591 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1167/2358] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:43:12 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:43:14.443992 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f727d66-aa5d-4f3f-a869-5e8c5dedab41 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:14.514927 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f727d66-aa5d-4f3f-a869-5e8c5dedab41 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:14.519471 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a2215b7-af73-4d4e-881f-4517788c8d03 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:14.520093 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a2215b7-af73-4d4e-881f-4517788c8d03 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:14.520370 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a2215b7-af73-4d4e-881f-4517788c8d03 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:14.520660 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a2215b7-af73-4d4e-881f-4517788c8d03 admin admin] PATH_INFO: `/v3/users/a968a29f3730486aa8ebb9e1fd182470` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:14.521782 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a2215b7-af73-4d4e-881f-4517788c8d03 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a968a29f3730486aa8ebb9e1fd182470)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:14.523854 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a2215b7-af73-4d4e-881f-4517788c8d03 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:14.538178 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:INSERT=3,keystone:SELECT=96,keystone:DELETE=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:43:14.576685 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-5a2215b7-af73-4d4e-881f-4517788c8d03 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a968a29f3730486aa8ebb9e1fd182470'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:14.585152 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-5a2215b7-af73-4d4e-881f-4517788c8d03 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a968a29f3730486aa8ebb9e1fd182470'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:14.599418 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-5a2215b7-af73-4d4e-881f-4517788c8d03 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a968a29f3730486aa8ebb9e1fd182470'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:14.600061 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-5a2215b7-af73-4d4e-881f-4517788c8d03 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a968a29f3730486aa8ebb9e1fd182470'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:14.607423 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-5a2215b7-af73-4d4e-881f-4517788c8d03 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a968a29f3730486aa8ebb9e1fd182470'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:14.614543 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1192/2359] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:43:14 2022] DELETE /identity/v3/users/a968a29f3730486aa8ebb9e1fd182470 => generated 0 bytes in 172 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:43:14.619205 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56fbb90b-900c-4ac7-b95d-51a4f993cb3f None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:14.619685 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56fbb90b-900c-4ac7-b95d-51a4f993cb3f None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:14.619773 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56fbb90b-900c-4ac7-b95d-51a4f993cb3f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:14.642432 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-56fbb90b-900c-4ac7-b95d-51a4f993cb3f None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:43:14.644020 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-56fbb90b-900c-4ac7-b95d-51a4f993cb3f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:43:14.678227 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:43:14.716786 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1168/2360] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:43:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:43:14.727692 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a2215b7-af73-4d4e-881f-4517788c8d03 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:14.729879 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-5a2215b7-af73-4d4e-881f-4517788c8d03 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:43:14.791809 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a2215b7-af73-4d4e-881f-4517788c8d03 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:14.795960 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0caf872-e9ba-4f13-9bc0-616e720ecd77 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:14.796098 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0caf872-e9ba-4f13-9bc0-616e720ecd77 service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:14.796382 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0caf872-e9ba-4f13-9bc0-616e720ecd77 service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:14.796673 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0caf872-e9ba-4f13-9bc0-616e720ecd77 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:14.806040 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0caf872-e9ba-4f13-9bc0-616e720ecd77 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:14.807493 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0caf872-e9ba-4f13-9bc0-616e720ecd77 service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:14.818040 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1193/2361] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:43:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:43:15.085339 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56fbb90b-900c-4ac7-b95d-51a4f993cb3f None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:15.086854 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-56fbb90b-900c-4ac7-b95d-51a4f993cb3f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:43:15.099062 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56fbb90b-900c-4ac7-b95d-51a4f993cb3f None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:15.103766 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc98d766-2dc6-4062-886c-20bf52e20481 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:15.104412 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc98d766-2dc6-4062-886c-20bf52e20481 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:15.104725 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc98d766-2dc6-4062-886c-20bf52e20481 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:15.105008 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc98d766-2dc6-4062-886c-20bf52e20481 admin admin] PATH_INFO: `/v3/projects/d3f8a3c6dad64c04a37c1f1abe51d6b8` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:15.106189 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc98d766-2dc6-4062-886c-20bf52e20481 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d3f8a3c6dad64c04a37c1f1abe51d6b8)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:15.108100 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc98d766-2dc6-4062-886c-20bf52e20481 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:15.156042 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-bc98d766-2dc6-4062-886c-20bf52e20481 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd3f8a3c6dad64c04a37c1f1abe51d6b8', 'request_id': 'req-bc98d766-2dc6-4062-886c-20bf52e20481'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:15.163342 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-bc98d766-2dc6-4062-886c-20bf52e20481 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd3f8a3c6dad64c04a37c1f1abe51d6b8', 'request_id': 'req-bc98d766-2dc6-4062-886c-20bf52e20481'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:15.168926 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-bc98d766-2dc6-4062-886c-20bf52e20481 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd3f8a3c6dad64c04a37c1f1abe51d6b8', 'request_id': 'req-bc98d766-2dc6-4062-886c-20bf52e20481'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:15.180166 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-bc98d766-2dc6-4062-886c-20bf52e20481 admin admin] The token cache is being invalidate because project d3f8a3c6dad64c04a37c1f1abe51d6b8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:43:15.180458 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-bc98d766-2dc6-4062-886c-20bf52e20481 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:15.181198 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-bc98d766-2dc6-4062-886c-20bf52e20481 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:15.182698 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1169/2362] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:43:15 2022] DELETE /identity/v3/projects/d3f8a3c6dad64c04a37c1f1abe51d6b8 => generated 0 bytes in 98 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:43:15.187587 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-563c3776-d6fc-4cd8-9772-c1568bdc6adb None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:15.187868 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-563c3776-d6fc-4cd8-9772-c1568bdc6adb None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:15.188140 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-563c3776-d6fc-4cd8-9772-c1568bdc6adb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:15.208271 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-563c3776-d6fc-4cd8-9772-c1568bdc6adb None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:43:15.209554 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-563c3776-d6fc-4cd8-9772-c1568bdc6adb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:43:15.270315 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1194/2363] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:43:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:43:15.277906 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8be6e54-b45a-4f82-8be2-9d16a07465ce None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:15.278211 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8be6e54-b45a-4f82-8be2-9d16a07465ce None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:15.278611 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8be6e54-b45a-4f82-8be2-9d16a07465ce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:15.300953 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-f8be6e54-b45a-4f82-8be2-9d16a07465ce None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:43:15.302392 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-f8be6e54-b45a-4f82-8be2-9d16a07465ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:43:15.313445 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1170/2364] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:43:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:43:15.318754 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-563c3776-d6fc-4cd8-9772-c1568bdc6adb None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:15.329740 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-563c3776-d6fc-4cd8-9772-c1568bdc6adb None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:15.334693 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d512e4a-34ab-4294-81c5-46a3aa7f5fee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:15.335303 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d512e4a-34ab-4294-81c5-46a3aa7f5fee admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:15.335585 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d512e4a-34ab-4294-81c5-46a3aa7f5fee admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:15.335861 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d512e4a-34ab-4294-81c5-46a3aa7f5fee admin admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:15.336507 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d512e4a-34ab-4294-81c5-46a3aa7f5fee admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:43:15.336843 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d512e4a-34ab-4294-81c5-46a3aa7f5fee admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:15.338708 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d512e4a-34ab-4294-81c5-46a3aa7f5fee admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:15.354391 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1195/2365] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Sep 25 04:43:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:43:15.355564 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8be6e54-b45a-4f82-8be2-9d16a07465ce None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:15.366605 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8be6e54-b45a-4f82-8be2-9d16a07465ce None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:15.371653 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22449d73-83e5-46ec-8816-c7a08e2e7c0e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:15.372269 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22449d73-83e5-46ec-8816-c7a08e2e7c0e admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:15.372602 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22449d73-83e5-46ec-8816-c7a08e2e7c0e admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:15.372973 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22449d73-83e5-46ec-8816-c7a08e2e7c0e admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:15.373620 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22449d73-83e5-46ec-8816-c7a08e2e7c0e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:15.375698 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22449d73-83e5-46ec-8816-c7a08e2e7c0e admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:15.408737 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1171/2366] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:43:15 2022] POST /identity/v3/projects => generated 365 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:43:15.412460 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d512e4a-34ab-4294-81c5-46a3aa7f5fee admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:15.426386 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d512e4a-34ab-4294-81c5-46a3aa7f5fee admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:15.449339 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:43:15.515988 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32cd2f5e-a2e2-4ef3-b942-22d586707a6c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:15.516957 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32cd2f5e-a2e2-4ef3-b942-22d586707a6c admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:15.517388 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32cd2f5e-a2e2-4ef3-b942-22d586707a6c admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:15.517810 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32cd2f5e-a2e2-4ef3-b942-22d586707a6c admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:15.518605 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32cd2f5e-a2e2-4ef3-b942-22d586707a6c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:15.521299 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32cd2f5e-a2e2-4ef3-b942-22d586707a6c admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:15.544681 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1196/2367] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:43:15 2022] POST /identity/v3/users => generated 283 bytes in 132 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:43:15.548312 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22449d73-83e5-46ec-8816-c7a08e2e7c0e admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:15.562607 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22449d73-83e5-46ec-8816-c7a08e2e7c0e admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:15.567630 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9afb534-55a0-404d-b2d7-e6c4c023b48a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:15.568422 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9afb534-55a0-404d-b2d7-e6c4c023b48a admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:15.568810 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9afb534-55a0-404d-b2d7-e6c4c023b48a admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:15.569153 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9afb534-55a0-404d-b2d7-e6c4c023b48a admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:15.569859 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9afb534-55a0-404d-b2d7-e6c4c023b48a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:15.571480 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9afb534-55a0-404d-b2d7-e6c4c023b48a admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:15.585302 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1172/2368] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:43:15 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 25 04:43:15.591569 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32cd2f5e-a2e2-4ef3-b942-22d586707a6c admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:15.607758 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32cd2f5e-a2e2-4ef3-b942-22d586707a6c admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:15.620711 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39d0bf2e-2d8d-4bd0-b26b-c82eac7bcd65 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:15.620711 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d0bf2e-2d8d-4bd0-b26b-c82eac7bcd65 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:15.620711 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d0bf2e-2d8d-4bd0-b26b-c82eac7bcd65 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:15.620711 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d0bf2e-2d8d-4bd0-b26b-c82eac7bcd65 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:15.621200 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39d0bf2e-2d8d-4bd0-b26b-c82eac7bcd65 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:15.628311 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39d0bf2e-2d8d-4bd0-b26b-c82eac7bcd65 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:15.646972 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1197/2369] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:43:15 2022] GET /identity/v3/roles => generated 1420 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:43:15.651318 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9afb534-55a0-404d-b2d7-e6c4c023b48a admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:15.664129 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9afb534-55a0-404d-b2d7-e6c4c023b48a admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:15.670784 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b367cb74-338f-459d-b270-9907e92f19c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:15.671509 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b367cb74-338f-459d-b270-9907e92f19c9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:15.672015 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b367cb74-338f-459d-b270-9907e92f19c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:15.672362 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b367cb74-338f-459d-b270-9907e92f19c9 admin admin] PATH_INFO: `/v3/projects/8c480f06a4504fcb84dd45884a40ab58/users/32c33257cb384f43b43089222203c72f/roles/8f6cc574056c4278bc9778f14dc4b411` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:15.688676 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b367cb74-338f-459d-b270-9907e92f19c9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8c480f06a4504fcb84dd45884a40ab58, user_id=32c33257cb384f43b43089222203c72f, role_id=8f6cc574056c4278bc9778f14dc4b411)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:15.691377 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b367cb74-338f-459d-b270-9907e92f19c9 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:15.704051 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1173/2370] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:43:15 2022] PUT /identity/v3/projects/8c480f06a4504fcb84dd45884a40ab58/users/32c33257cb384f43b43089222203c72f/roles/8f6cc574056c4278bc9778f14dc4b411 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:43:15.714965 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ddfb987-a7ea-48bb-b54a-fb761ad2af2c None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:15.715333 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ddfb987-a7ea-48bb-b54a-fb761ad2af2c None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:15.715674 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ddfb987-a7ea-48bb-b54a-fb761ad2af2c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:15.754710 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-8ddfb987-a7ea-48bb-b54a-fb761ad2af2c None None] MFA Rules not processed for user `32c33257cb384f43b43089222203c72f`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:43:15.755929 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-8ddfb987-a7ea-48bb-b54a-fb761ad2af2c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:43:15.832804 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1198/2371] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:43:15 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:43:15.835949 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b367cb74-338f-459d-b270-9907e92f19c9 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:15.912292 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:43:15.918813 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b367cb74-338f-459d-b270-9907e92f19c9 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:15.928679 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-125667fe-2021-4c15-9683-3fe341bf128e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:15.929703 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-125667fe-2021-4c15-9683-3fe341bf128e admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:15.930227 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-125667fe-2021-4c15-9683-3fe341bf128e admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:15.930831 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-125667fe-2021-4c15-9683-3fe341bf128e admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:15.931492 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-125667fe-2021-4c15-9683-3fe341bf128e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:15.933533 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-125667fe-2021-4c15-9683-3fe341bf128e admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:15.961534 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1174/2372] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:43:15 2022] POST /identity/v3/projects => generated 363 bytes in 126 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:43:15.965923 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ddfb987-a7ea-48bb-b54a-fb761ad2af2c None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:15.977651 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ddfb987-a7ea-48bb-b54a-fb761ad2af2c None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:16.034519 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ef97463-a40d-42d6-90ce-ba97d87c340b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:16.035932 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ef97463-a40d-42d6-90ce-ba97d87c340b admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:16.036330 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ef97463-a40d-42d6-90ce-ba97d87c340b admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:16.036622 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ef97463-a40d-42d6-90ce-ba97d87c340b admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:16.037272 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ef97463-a40d-42d6-90ce-ba97d87c340b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:16.039191 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ef97463-a40d-42d6-90ce-ba97d87c340b admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:16.059725 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1199/2373] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:43:15 2022] POST /identity/v3/users => generated 288 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 04:43:16.063526 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-125667fe-2021-4c15-9683-3fe341bf128e admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:16.075702 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-125667fe-2021-4c15-9683-3fe341bf128e admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:16.081203 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fd3e6ce-2bc2-4257-8d56-d422af6277d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:16.082468 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fd3e6ce-2bc2-4257-8d56-d422af6277d3 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:16.083205 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fd3e6ce-2bc2-4257-8d56-d422af6277d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:16.083503 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fd3e6ce-2bc2-4257-8d56-d422af6277d3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:16.084153 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fd3e6ce-2bc2-4257-8d56-d422af6277d3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:16.085802 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fd3e6ce-2bc2-4257-8d56-d422af6277d3 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:16.096899 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1175/2374] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:43:16 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:43:16.102034 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ef97463-a40d-42d6-90ce-ba97d87c340b admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:16.112374 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ef97463-a40d-42d6-90ce-ba97d87c340b admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:16.118062 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-119226a4-dc91-43d5-bb0b-38b626cb6883 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:16.118751 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-119226a4-dc91-43d5-bb0b-38b626cb6883 admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:16.119160 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-119226a4-dc91-43d5-bb0b-38b626cb6883 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:16.119441 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-119226a4-dc91-43d5-bb0b-38b626cb6883 admin admin] PATH_INFO: `/v3/projects/e74befd9fb134cc0a82fd2ce53c9c24a/users/8436d5cdddfb41e994c77c7008c71988/roles/c51fe7c5ad804040879acfb54269455f` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:16.136789 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-119226a4-dc91-43d5-bb0b-38b626cb6883 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e74befd9fb134cc0a82fd2ce53c9c24a, user_id=8436d5cdddfb41e994c77c7008c71988, role_id=c51fe7c5ad804040879acfb54269455f)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:16.139500 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-119226a4-dc91-43d5-bb0b-38b626cb6883 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:16.142806 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:43:16.154854 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1200/2375] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:43:16 2022] PUT /identity/v3/projects/e74befd9fb134cc0a82fd2ce53c9c24a/users/8436d5cdddfb41e994c77c7008c71988/roles/c51fe7c5ad804040879acfb54269455f => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:43:16.159008 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fd3e6ce-2bc2-4257-8d56-d422af6277d3 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:16.169903 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fd3e6ce-2bc2-4257-8d56-d422af6277d3 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:16.245956 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1b91526-a863-490b-a602-39d9503cc223 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:16.246669 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1b91526-a863-490b-a602-39d9503cc223 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:16.247021 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1b91526-a863-490b-a602-39d9503cc223 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:16.247366 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1b91526-a863-490b-a602-39d9503cc223 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:16.248083 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1b91526-a863-490b-a602-39d9503cc223 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:16.249801 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1b91526-a863-490b-a602-39d9503cc223 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:16.263057 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1176/2376] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:43:16 2022] GET /identity/v3/roles => generated 1420 bytes in 104 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:43:16.267642 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-119226a4-dc91-43d5-bb0b-38b626cb6883 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:16.280879 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-119226a4-dc91-43d5-bb0b-38b626cb6883 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:16.286778 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-997e54f8-0b82-4a63-aca7-164ad27c27e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:16.287452 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-997e54f8-0b82-4a63-aca7-164ad27c27e4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:16.287730 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-997e54f8-0b82-4a63-aca7-164ad27c27e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:16.288002 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-997e54f8-0b82-4a63-aca7-164ad27c27e4 admin admin] PATH_INFO: `/v3/projects/e74befd9fb134cc0a82fd2ce53c9c24a/users/8436d5cdddfb41e994c77c7008c71988/roles/c51fe7c5ad804040879acfb54269455f` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:16.289971 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-997e54f8-0b82-4a63-aca7-164ad27c27e4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e74befd9fb134cc0a82fd2ce53c9c24a, user_id=8436d5cdddfb41e994c77c7008c71988, role_id=c51fe7c5ad804040879acfb54269455f)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:16.292707 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-997e54f8-0b82-4a63-aca7-164ad27c27e4 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:16.304367 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1201/2377] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:43:16 2022] PUT /identity/v3/projects/e74befd9fb134cc0a82fd2ce53c9c24a/users/8436d5cdddfb41e994c77c7008c71988/roles/c51fe7c5ad804040879acfb54269455f => generated 0 bytes in 37 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:43:16.316776 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2da67dff-03bb-47ea-af59-f4a83c9a54df None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:16.316991 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2da67dff-03bb-47ea-af59-f4a83c9a54df None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:16.317266 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2da67dff-03bb-47ea-af59-f4a83c9a54df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:16.348279 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-2da67dff-03bb-47ea-af59-f4a83c9a54df None None] MFA Rules not processed for user `8436d5cdddfb41e994c77c7008c71988`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:43:16.349635 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-2da67dff-03bb-47ea-af59-f4a83c9a54df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:43:16.419465 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1177/2378] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:43:16 2022] POST /identity/v3/auth/tokens => generated 3307 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:43:16.430773 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-997e54f8-0b82-4a63-aca7-164ad27c27e4 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:16.432482 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-997e54f8-0b82-4a63-aca7-164ad27c27e4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:43:16.492509 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-997e54f8-0b82-4a63-aca7-164ad27c27e4 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:16.496400 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b67ca30-96d0-449b-88eb-cb5c51288af9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ffbc136c77b84e8c87128f7247f4ec03', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:16.497079 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b67ca30-96d0-449b-88eb-cb5c51288af9 service cinder] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:16.497362 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b67ca30-96d0-449b-88eb-cb5c51288af9 service cinder] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:16.497638 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b67ca30-96d0-449b-88eb-cb5c51288af9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:16.507009 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b67ca30-96d0-449b-88eb-cb5c51288af9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:16.508559 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b67ca30-96d0-449b-88eb-cb5c51288af9 service cinder] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:16.518729 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1202/2379] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:43:16 2022] GET /identity/v3/auth/tokens => generated 3307 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:43:16.601138 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2da67dff-03bb-47ea-af59-f4a83c9a54df None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:16.611819 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2da67dff-03bb-47ea-af59-f4a83c9a54df None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:16.616175 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6b1523c-a59a-4ea8-b4c5-dbf0c8bd146c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ffbc136c77b84e8c87128f7247f4ec03', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:16.616852 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b1523c-a59a-4ea8-b4c5-dbf0c8bd146c service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:16.617136 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b1523c-a59a-4ea8-b4c5-dbf0c8bd146c service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:16.617408 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b1523c-a59a-4ea8-b4c5-dbf0c8bd146c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:16.691044 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:43:16.707828 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6b1523c-a59a-4ea8-b4c5-dbf0c8bd146c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:16.709385 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6b1523c-a59a-4ea8-b4c5-dbf0c8bd146c service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:16.724737 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1178/2380] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:43:16 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 124 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:43:18.519597 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d43419-40fb-4385-912c-5457c3f68558 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:18.520149 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d43419-40fb-4385-912c-5457c3f68558 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:18.520149 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d43419-40fb-4385-912c-5457c3f68558 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:18.541962 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-90d43419-40fb-4385-912c-5457c3f68558 None None] MFA Rules not processed for user `050b3aab44bb4b17aa4a1ba7cccdb231`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:43:18.543303 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-90d43419-40fb-4385-912c-5457c3f68558 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:43:18.616355 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1203/2381] 10.222.0.25 () {58 vars in 967 bytes} [Sun Sep 25 04:43:18 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:43:18.666705 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6b1523c-a59a-4ea8-b4c5-dbf0c8bd146c service cinder] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:18.668380 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-a6b1523c-a59a-4ea8-b4c5-dbf0c8bd146c service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:43:18.756467 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6b1523c-a59a-4ea8-b4c5-dbf0c8bd146c service cinder] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:18.761332 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dc6a2f7-8951-4fa9-9f13-d3df8855d39c service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '23ea88f46e7740d295148a93ade29dd9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:18.762147 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dc6a2f7-8951-4fa9-9f13-d3df8855d39c service swift] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:18.762609 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dc6a2f7-8951-4fa9-9f13-d3df8855d39c service swift] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:18.763035 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dc6a2f7-8951-4fa9-9f13-d3df8855d39c service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:18.774116 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dc6a2f7-8951-4fa9-9f13-d3df8855d39c service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:18.775730 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dc6a2f7-8951-4fa9-9f13-d3df8855d39c service swift] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:18.786293 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1179/2382] 10.222.0.25 () {60 vars in 1384 bytes} [Sun Sep 25 04:43:18 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 120 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 25 04:43:22.367824 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90d43419-40fb-4385-912c-5457c3f68558 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:22.369439 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-90d43419-40fb-4385-912c-5457c3f68558 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:43:22.472819 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90d43419-40fb-4385-912c-5457c3f68558 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:22.478340 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b18a4159-8a2f-465e-b548-2a369f99d5d0 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '676d04ef5a984edcaad63d9156568660', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:22.479481 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b18a4159-8a2f-465e-b548-2a369f99d5d0 service placement] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:22.480010 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b18a4159-8a2f-465e-b548-2a369f99d5d0 service placement] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:22.480510 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b18a4159-8a2f-465e-b548-2a369f99d5d0 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:22.482934 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-b18a4159-8a2f-465e-b548-2a369f99d5d0 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:43:22.495864 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b18a4159-8a2f-465e-b548-2a369f99d5d0 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:22.497571 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b18a4159-8a2f-465e-b548-2a369f99d5d0 service placement] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:22.508118 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1204/2383] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:43:22 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 141 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:43:25.603564 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dc6a2f7-8951-4fa9-9f13-d3df8855d39c service swift] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:25.605062 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-1dc6a2f7-8951-4fa9-9f13-d3df8855d39c service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:43:25.695702 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dc6a2f7-8951-4fa9-9f13-d3df8855d39c service swift] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:25.701174 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95a48598-76eb-4015-9053-4c9bab6e19f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:25.701807 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a48598-76eb-4015-9053-4c9bab6e19f1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:25.702093 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a48598-76eb-4015-9053-4c9bab6e19f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:25.702363 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a48598-76eb-4015-9053-4c9bab6e19f1 admin admin] PATH_INFO: `/v3/users/5902801c596048389599b2b18df16dd4` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:25.703489 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95a48598-76eb-4015-9053-4c9bab6e19f1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5902801c596048389599b2b18df16dd4)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:25.705449 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95a48598-76eb-4015-9053-4c9bab6e19f1 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:25.761014 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-95a48598-76eb-4015-9053-4c9bab6e19f1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5902801c596048389599b2b18df16dd4'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:25.768809 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-95a48598-76eb-4015-9053-4c9bab6e19f1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5902801c596048389599b2b18df16dd4'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:25.781692 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-95a48598-76eb-4015-9053-4c9bab6e19f1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5902801c596048389599b2b18df16dd4'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:25.782295 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-95a48598-76eb-4015-9053-4c9bab6e19f1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5902801c596048389599b2b18df16dd4'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:25.790119 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-95a48598-76eb-4015-9053-4c9bab6e19f1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5902801c596048389599b2b18df16dd4'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:25.791675 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:43:25.797795 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1180/2384] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:43:25 2022] DELETE /identity/v3/users/5902801c596048389599b2b18df16dd4 => generated 0 bytes in 195 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:43:25.981145 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b18a4159-8a2f-465e-b548-2a369f99d5d0 service placement] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:25.992864 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b18a4159-8a2f-465e-b548-2a369f99d5d0 service placement] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:26.016246 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:43:26.066244 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6229f09-9986-4d22-9db9-b2c60491dbde admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:26.066779 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6229f09-9986-4d22-9db9-b2c60491dbde admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:26.067060 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6229f09-9986-4d22-9db9-b2c60491dbde admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:26.067335 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6229f09-9986-4d22-9db9-b2c60491dbde admin admin] PATH_INFO: `/v3/projects/b5c1180472fb4243bd45e44914a46918` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:26.068466 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6229f09-9986-4d22-9db9-b2c60491dbde admin admin] RBAC: Authorizing `identity:delete_project(project_id=b5c1180472fb4243bd45e44914a46918)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:26.070441 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6229f09-9986-4d22-9db9-b2c60491dbde admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:26.118070 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-f6229f09-9986-4d22-9db9-b2c60491dbde admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b5c1180472fb4243bd45e44914a46918', 'request_id': 'req-f6229f09-9986-4d22-9db9-b2c60491dbde'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:26.125291 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-f6229f09-9986-4d22-9db9-b2c60491dbde admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b5c1180472fb4243bd45e44914a46918', 'request_id': 'req-f6229f09-9986-4d22-9db9-b2c60491dbde'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:26.131267 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-f6229f09-9986-4d22-9db9-b2c60491dbde admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b5c1180472fb4243bd45e44914a46918', 'request_id': 'req-f6229f09-9986-4d22-9db9-b2c60491dbde'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:26.142382 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-f6229f09-9986-4d22-9db9-b2c60491dbde admin admin] The token cache is being invalidate because project b5c1180472fb4243bd45e44914a46918 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81434) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:43:26.142749 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-f6229f09-9986-4d22-9db9-b2c60491dbde admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:26.143495 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-f6229f09-9986-4d22-9db9-b2c60491dbde admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:26.145385 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1205/2385] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:43:25 2022] DELETE /identity/v3/projects/b5c1180472fb4243bd45e44914a46918 => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 04:43:26.149493 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95a48598-76eb-4015-9053-4c9bab6e19f1 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:26.150963 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-95a48598-76eb-4015-9053-4c9bab6e19f1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:43:26.164623 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95a48598-76eb-4015-9053-4c9bab6e19f1 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:26.223348 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a95ec28e-13f5-4717-90a9-75cf34dbb208 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:26.223980 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a95ec28e-13f5-4717-90a9-75cf34dbb208 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:26.224264 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a95ec28e-13f5-4717-90a9-75cf34dbb208 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:26.224541 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a95ec28e-13f5-4717-90a9-75cf34dbb208 admin admin] PATH_INFO: `/v3/users/40bb42d222744b17815207ad07c36c2e` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:26.225710 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a95ec28e-13f5-4717-90a9-75cf34dbb208 admin admin] RBAC: Authorizing `identity:delete_user(user_id=40bb42d222744b17815207ad07c36c2e)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:26.227853 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a95ec28e-13f5-4717-90a9-75cf34dbb208 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:26.274454 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-a95ec28e-13f5-4717-90a9-75cf34dbb208 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '40bb42d222744b17815207ad07c36c2e'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:26.282145 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-a95ec28e-13f5-4717-90a9-75cf34dbb208 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '40bb42d222744b17815207ad07c36c2e'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:26.295659 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-a95ec28e-13f5-4717-90a9-75cf34dbb208 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '40bb42d222744b17815207ad07c36c2e'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:26.296336 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-a95ec28e-13f5-4717-90a9-75cf34dbb208 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '40bb42d222744b17815207ad07c36c2e'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:26.304486 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-a95ec28e-13f5-4717-90a9-75cf34dbb208 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '40bb42d222744b17815207ad07c36c2e'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:26.310744 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1181/2386] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:43:26 2022] DELETE /identity/v3/users/40bb42d222744b17815207ad07c36c2e => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:43:26.493772 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6229f09-9986-4d22-9db9-b2c60491dbde admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:26.495302 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-f6229f09-9986-4d22-9db9-b2c60491dbde admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:43:26.510002 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6229f09-9986-4d22-9db9-b2c60491dbde admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:26.579804 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f89ab63-cb4b-405f-a2e8-8e88093a17a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:26.581194 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f89ab63-cb4b-405f-a2e8-8e88093a17a8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:26.582036 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f89ab63-cb4b-405f-a2e8-8e88093a17a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:26.582483 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f89ab63-cb4b-405f-a2e8-8e88093a17a8 admin admin] PATH_INFO: `/v3/projects/388ceb38ad354423992a7cc1eae04f44` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:26.583888 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f89ab63-cb4b-405f-a2e8-8e88093a17a8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=388ceb38ad354423992a7cc1eae04f44)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:26.585865 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f89ab63-cb4b-405f-a2e8-8e88093a17a8 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:26.637128 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-7f89ab63-cb4b-405f-a2e8-8e88093a17a8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '388ceb38ad354423992a7cc1eae04f44', 'request_id': 'req-7f89ab63-cb4b-405f-a2e8-8e88093a17a8'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:26.639760 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=87,keystone:DELETE=11,keystone:INSERT=2 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:43:26.647561 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-7f89ab63-cb4b-405f-a2e8-8e88093a17a8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '388ceb38ad354423992a7cc1eae04f44', 'request_id': 'req-7f89ab63-cb4b-405f-a2e8-8e88093a17a8'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:26.654819 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-7f89ab63-cb4b-405f-a2e8-8e88093a17a8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '388ceb38ad354423992a7cc1eae04f44', 'request_id': 'req-7f89ab63-cb4b-405f-a2e8-8e88093a17a8'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:26.667061 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-7f89ab63-cb4b-405f-a2e8-8e88093a17a8 admin admin] The token cache is being invalidate because project 388ceb38ad354423992a7cc1eae04f44 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81434) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:43:26.667358 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-7f89ab63-cb4b-405f-a2e8-8e88093a17a8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:26.668032 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-7f89ab63-cb4b-405f-a2e8-8e88093a17a8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:26.669670 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1206/2387] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:43:26 2022] DELETE /identity/v3/projects/388ceb38ad354423992a7cc1eae04f44 => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:43:28.515839 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a95ec28e-13f5-4717-90a9-75cf34dbb208 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:28.517292 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-a95ec28e-13f5-4717-90a9-75cf34dbb208 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:43:28.531161 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a95ec28e-13f5-4717-90a9-75cf34dbb208 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:28.587211 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8e2c732-1ea9-49b9-8d0e-1aad2bb66eaa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:28.588100 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8e2c732-1ea9-49b9-8d0e-1aad2bb66eaa admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:28.588384 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8e2c732-1ea9-49b9-8d0e-1aad2bb66eaa admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:28.588679 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8e2c732-1ea9-49b9-8d0e-1aad2bb66eaa admin admin] PATH_INFO: `/v3/users/32c33257cb384f43b43089222203c72f` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:28.589814 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8e2c732-1ea9-49b9-8d0e-1aad2bb66eaa admin admin] RBAC: Authorizing `identity:delete_user(user_id=32c33257cb384f43b43089222203c72f)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:28.591727 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8e2c732-1ea9-49b9-8d0e-1aad2bb66eaa admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:28.614433 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:43:28.641387 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-f8e2c732-1ea9-49b9-8d0e-1aad2bb66eaa admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '32c33257cb384f43b43089222203c72f'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:28.649237 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-f8e2c732-1ea9-49b9-8d0e-1aad2bb66eaa admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '32c33257cb384f43b43089222203c72f'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:28.661790 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-f8e2c732-1ea9-49b9-8d0e-1aad2bb66eaa admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '32c33257cb384f43b43089222203c72f'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:28.662425 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-f8e2c732-1ea9-49b9-8d0e-1aad2bb66eaa admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '32c33257cb384f43b43089222203c72f'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:28.669421 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-f8e2c732-1ea9-49b9-8d0e-1aad2bb66eaa admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '32c33257cb384f43b43089222203c72f'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:28.675843 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1182/2388] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:43:28 2022] DELETE /identity/v3/users/32c33257cb384f43b43089222203c72f => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:43:28.680460 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08d7546e-ecbe-4c60-9210-8ed7c5de7675 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:28.680972 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08d7546e-ecbe-4c60-9210-8ed7c5de7675 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:28.681073 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08d7546e-ecbe-4c60-9210-8ed7c5de7675 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:28.700623 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-08d7546e-ecbe-4c60-9210-8ed7c5de7675 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:43:28.701845 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-08d7546e-ecbe-4c60-9210-8ed7c5de7675 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:43:28.760943 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1207/2389] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:43:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:43:28.770702 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8e2c732-1ea9-49b9-8d0e-1aad2bb66eaa admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:28.772102 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-f8e2c732-1ea9-49b9-8d0e-1aad2bb66eaa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:43:28.829463 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8e2c732-1ea9-49b9-8d0e-1aad2bb66eaa admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:28.833327 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7f4f61e-3fef-4cd2-8ef9-b205f8a19b4f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:28.833933 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7f4f61e-3fef-4cd2-8ef9-b205f8a19b4f service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:28.834212 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7f4f61e-3fef-4cd2-8ef9-b205f8a19b4f service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:28.834530 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7f4f61e-3fef-4cd2-8ef9-b205f8a19b4f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:28.843706 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7f4f61e-3fef-4cd2-8ef9-b205f8a19b4f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:28.845178 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7f4f61e-3fef-4cd2-8ef9-b205f8a19b4f service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:28.854717 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1183/2390] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:43:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:43:29.125117 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08d7546e-ecbe-4c60-9210-8ed7c5de7675 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:29.127119 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-08d7546e-ecbe-4c60-9210-8ed7c5de7675 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:43:29.143568 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08d7546e-ecbe-4c60-9210-8ed7c5de7675 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:29.148508 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc3e355a-12b4-4039-acd6-907c223de66c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:29.149248 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc3e355a-12b4-4039-acd6-907c223de66c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:29.149595 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc3e355a-12b4-4039-acd6-907c223de66c admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:29.149960 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc3e355a-12b4-4039-acd6-907c223de66c admin admin] PATH_INFO: `/v3/projects/8c480f06a4504fcb84dd45884a40ab58` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:29.151159 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc3e355a-12b4-4039-acd6-907c223de66c admin admin] RBAC: Authorizing `identity:delete_project(project_id=8c480f06a4504fcb84dd45884a40ab58)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:29.153180 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc3e355a-12b4-4039-acd6-907c223de66c admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:29.206990 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-dc3e355a-12b4-4039-acd6-907c223de66c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8c480f06a4504fcb84dd45884a40ab58', 'request_id': 'req-dc3e355a-12b4-4039-acd6-907c223de66c'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:29.216125 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-dc3e355a-12b4-4039-acd6-907c223de66c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8c480f06a4504fcb84dd45884a40ab58', 'request_id': 'req-dc3e355a-12b4-4039-acd6-907c223de66c'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:29.224195 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-dc3e355a-12b4-4039-acd6-907c223de66c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8c480f06a4504fcb84dd45884a40ab58', 'request_id': 'req-dc3e355a-12b4-4039-acd6-907c223de66c'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:29.236347 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-dc3e355a-12b4-4039-acd6-907c223de66c admin admin] The token cache is being invalidate because project 8c480f06a4504fcb84dd45884a40ab58 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81434) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:43:29.236724 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-dc3e355a-12b4-4039-acd6-907c223de66c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:29.237398 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-dc3e355a-12b4-4039-acd6-907c223de66c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:29.239913 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1208/2391] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:43:29 2022] DELETE /identity/v3/projects/8c480f06a4504fcb84dd45884a40ab58 => generated 0 bytes in 115 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:43:29.243659 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7f4f61e-3fef-4cd2-8ef9-b205f8a19b4f service neutron] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:29.253635 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7f4f61e-3fef-4cd2-8ef9-b205f8a19b4f service neutron] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:29.317156 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23b592d3-79b2-48bd-8b07-7fdd465198fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:29.317833 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23b592d3-79b2-48bd-8b07-7fdd465198fd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:29.318115 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23b592d3-79b2-48bd-8b07-7fdd465198fd admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:29.318390 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23b592d3-79b2-48bd-8b07-7fdd465198fd admin admin] PATH_INFO: `/v3/users/8436d5cdddfb41e994c77c7008c71988` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:29.320091 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23b592d3-79b2-48bd-8b07-7fdd465198fd admin admin] RBAC: Authorizing `identity:delete_user(user_id=8436d5cdddfb41e994c77c7008c71988)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:29.322073 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23b592d3-79b2-48bd-8b07-7fdd465198fd admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:29.381723 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:DELETE=14,keystone:SELECT=85,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:43:29.388170 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-23b592d3-79b2-48bd-8b07-7fdd465198fd admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8436d5cdddfb41e994c77c7008c71988'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:29.397239 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-23b592d3-79b2-48bd-8b07-7fdd465198fd admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8436d5cdddfb41e994c77c7008c71988'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:29.410524 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-23b592d3-79b2-48bd-8b07-7fdd465198fd admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8436d5cdddfb41e994c77c7008c71988'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:29.411297 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-23b592d3-79b2-48bd-8b07-7fdd465198fd admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8436d5cdddfb41e994c77c7008c71988'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:29.418148 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-23b592d3-79b2-48bd-8b07-7fdd465198fd admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8436d5cdddfb41e994c77c7008c71988'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:29.423978 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1184/2392] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:43:29 2022] DELETE /identity/v3/users/8436d5cdddfb41e994c77c7008c71988 => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:43:29.697904 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc3e355a-12b4-4039-acd6-907c223de66c admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:29.699882 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-dc3e355a-12b4-4039-acd6-907c223de66c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:43:29.719185 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc3e355a-12b4-4039-acd6-907c223de66c admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:29.776503 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f998e040-5b5b-42cc-823a-c1d59bc8bbfd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:29.777171 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f998e040-5b5b-42cc-823a-c1d59bc8bbfd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:29.777452 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f998e040-5b5b-42cc-823a-c1d59bc8bbfd admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:29.777725 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f998e040-5b5b-42cc-823a-c1d59bc8bbfd admin admin] PATH_INFO: `/v3/projects/e74befd9fb134cc0a82fd2ce53c9c24a` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:29.778836 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f998e040-5b5b-42cc-823a-c1d59bc8bbfd admin admin] RBAC: Authorizing `identity:delete_project(project_id=e74befd9fb134cc0a82fd2ce53c9c24a)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:29.780761 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f998e040-5b5b-42cc-823a-c1d59bc8bbfd admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:29.787134 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:DELETE=10,keystone:SELECT=89,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:43:29.829397 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-f998e040-5b5b-42cc-823a-c1d59bc8bbfd admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e74befd9fb134cc0a82fd2ce53c9c24a', 'request_id': 'req-f998e040-5b5b-42cc-823a-c1d59bc8bbfd'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:29.838347 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-f998e040-5b5b-42cc-823a-c1d59bc8bbfd admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e74befd9fb134cc0a82fd2ce53c9c24a', 'request_id': 'req-f998e040-5b5b-42cc-823a-c1d59bc8bbfd'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:29.843936 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-f998e040-5b5b-42cc-823a-c1d59bc8bbfd admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e74befd9fb134cc0a82fd2ce53c9c24a', 'request_id': 'req-f998e040-5b5b-42cc-823a-c1d59bc8bbfd'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:29.855323 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-f998e040-5b5b-42cc-823a-c1d59bc8bbfd admin admin] The token cache is being invalidate because project e74befd9fb134cc0a82fd2ce53c9c24a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81434) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:43:29.855614 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-f998e040-5b5b-42cc-823a-c1d59bc8bbfd admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:29.856259 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-f998e040-5b5b-42cc-823a-c1d59bc8bbfd admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:29.857830 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1209/2393] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:43:29 2022] DELETE /identity/v3/projects/e74befd9fb134cc0a82fd2ce53c9c24a => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:43:29.862681 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f902eda-c803-4fa2-b8f6-a6e77d237a61 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:29.863194 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f902eda-c803-4fa2-b8f6-a6e77d237a61 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:29.863650 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f902eda-c803-4fa2-b8f6-a6e77d237a61 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:29.883625 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-4f902eda-c803-4fa2-b8f6-a6e77d237a61 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:43:29.885012 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-4f902eda-c803-4fa2-b8f6-a6e77d237a61 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:43:29.947446 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1185/2394] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:43:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:43:29.955369 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b86506c8-d662-438d-93ee-ff0fe8051031 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:29.955655 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b86506c8-d662-438d-93ee-ff0fe8051031 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:29.955926 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b86506c8-d662-438d-93ee-ff0fe8051031 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:29.976990 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-b86506c8-d662-438d-93ee-ff0fe8051031 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:43:29.978202 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-b86506c8-d662-438d-93ee-ff0fe8051031 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:43:29.986090 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1210/2395] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:43:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:43:29.991146 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f902eda-c803-4fa2-b8f6-a6e77d237a61 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:30.000859 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f902eda-c803-4fa2-b8f6-a6e77d237a61 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:30.005675 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fcefcfa-8e20-4e99-9738-d06cb11b12b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:30.006261 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fcefcfa-8e20-4e99-9738-d06cb11b12b1 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:30.006540 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fcefcfa-8e20-4e99-9738-d06cb11b12b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:30.006808 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fcefcfa-8e20-4e99-9738-d06cb11b12b1 admin admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:30.007446 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fcefcfa-8e20-4e99-9738-d06cb11b12b1 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:43:30.007745 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fcefcfa-8e20-4e99-9738-d06cb11b12b1 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:30.009366 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fcefcfa-8e20-4e99-9738-d06cb11b12b1 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:30.020952 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1186/2396] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Sep 25 04:43:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:43:30.025239 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b86506c8-d662-438d-93ee-ff0fe8051031 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:30.035284 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b86506c8-d662-438d-93ee-ff0fe8051031 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:30.039964 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-915d71a3-27cc-48dc-99f1-06cf6e99c40e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:30.040562 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-915d71a3-27cc-48dc-99f1-06cf6e99c40e admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:30.040870 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-915d71a3-27cc-48dc-99f1-06cf6e99c40e admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:30.041140 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-915d71a3-27cc-48dc-99f1-06cf6e99c40e admin admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:30.041776 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-915d71a3-27cc-48dc-99f1-06cf6e99c40e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:30.043662 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-915d71a3-27cc-48dc-99f1-06cf6e99c40e admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:30.070689 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1211/2397] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:43:30 2022] POST /identity/v3/projects => generated 367 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 04:43:30.074874 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fcefcfa-8e20-4e99-9738-d06cb11b12b1 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:30.085551 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fcefcfa-8e20-4e99-9738-d06cb11b12b1 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:30.140808 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0304101a-35ab-4164-8c8b-f89524c25bc5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:30.141371 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0304101a-35ab-4164-8c8b-f89524c25bc5 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:30.141648 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0304101a-35ab-4164-8c8b-f89524c25bc5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:30.141917 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0304101a-35ab-4164-8c8b-f89524c25bc5 admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:30.142548 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0304101a-35ab-4164-8c8b-f89524c25bc5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:30.145015 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0304101a-35ab-4164-8c8b-f89524c25bc5 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:30.166630 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1187/2398] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:43:30 2022] POST /identity/v3/users => generated 284 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 04:43:30.170694 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-915d71a3-27cc-48dc-99f1-06cf6e99c40e admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:30.180825 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-915d71a3-27cc-48dc-99f1-06cf6e99c40e admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:30.185431 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33b87238-8510-4d79-b72e-007853dbc7e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:30.186020 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33b87238-8510-4d79-b72e-007853dbc7e3 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:30.186302 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33b87238-8510-4d79-b72e-007853dbc7e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:30.186572 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33b87238-8510-4d79-b72e-007853dbc7e3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:30.187206 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33b87238-8510-4d79-b72e-007853dbc7e3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:30.188800 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33b87238-8510-4d79-b72e-007853dbc7e3 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:30.198769 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1212/2399] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:43:30 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:43:30.202445 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0304101a-35ab-4164-8c8b-f89524c25bc5 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:30.213599 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0304101a-35ab-4164-8c8b-f89524c25bc5 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:30.218327 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acf32841-8df0-4ca6-b3f7-b37b596518db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:30.219511 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf32841-8df0-4ca6-b3f7-b37b596518db admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:30.219511 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf32841-8df0-4ca6-b3f7-b37b596518db admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:30.219511 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf32841-8df0-4ca6-b3f7-b37b596518db admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:30.220069 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acf32841-8df0-4ca6-b3f7-b37b596518db admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:30.221692 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acf32841-8df0-4ca6-b3f7-b37b596518db admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:30.232905 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1188/2400] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:43:30 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:43:30.237083 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33b87238-8510-4d79-b72e-007853dbc7e3 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:30.247195 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33b87238-8510-4d79-b72e-007853dbc7e3 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:30.251757 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6370961-5934-4143-acad-38e20ad9a666 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:30.252393 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6370961-5934-4143-acad-38e20ad9a666 admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:30.252706 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6370961-5934-4143-acad-38e20ad9a666 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:30.252983 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6370961-5934-4143-acad-38e20ad9a666 admin admin] PATH_INFO: `/v3/projects/18bad7a155c14488b59ca7a55a7ed0bf/users/faa280fa43674f7c8bd538fd18e9b6f7/roles/8f6cc574056c4278bc9778f14dc4b411` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:30.268337 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6370961-5934-4143-acad-38e20ad9a666 admin admin] RBAC: Authorizing `identity:create_grant(project_id=18bad7a155c14488b59ca7a55a7ed0bf, user_id=faa280fa43674f7c8bd538fd18e9b6f7, role_id=8f6cc574056c4278bc9778f14dc4b411)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:30.271745 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6370961-5934-4143-acad-38e20ad9a666 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:30.285724 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1213/2401] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:43:30 2022] PUT /identity/v3/projects/18bad7a155c14488b59ca7a55a7ed0bf/users/faa280fa43674f7c8bd538fd18e9b6f7/roles/8f6cc574056c4278bc9778f14dc4b411 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:43:30.296945 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a83aa130-65d6-41a4-8a28-56305e3eeeff None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:30.297230 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a83aa130-65d6-41a4-8a28-56305e3eeeff None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:30.297503 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a83aa130-65d6-41a4-8a28-56305e3eeeff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:30.325281 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-a83aa130-65d6-41a4-8a28-56305e3eeeff None None] MFA Rules not processed for user `faa280fa43674f7c8bd538fd18e9b6f7`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:43:30.326983 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-a83aa130-65d6-41a4-8a28-56305e3eeeff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:43:30.340585 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=92,keystone:DELETE=4,keystone:INSERT=4 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:43:30.394931 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1189/2402] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:43:30 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:43:30.399531 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6370961-5934-4143-acad-38e20ad9a666 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:30.465765 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6370961-5934-4143-acad-38e20ad9a666 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:30.470577 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f2ac242-5123-428f-b8b6-7849365ce194 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:30.471200 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f2ac242-5123-428f-b8b6-7849365ce194 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:30.471482 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f2ac242-5123-428f-b8b6-7849365ce194 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:30.471760 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f2ac242-5123-428f-b8b6-7849365ce194 admin admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:30.472510 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f2ac242-5123-428f-b8b6-7849365ce194 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:30.474485 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f2ac242-5123-428f-b8b6-7849365ce194 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:30.504108 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1214/2403] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:43:30 2022] POST /identity/v3/projects => generated 369 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:43:30.508022 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a83aa130-65d6-41a4-8a28-56305e3eeeff None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:30.519640 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a83aa130-65d6-41a4-8a28-56305e3eeeff None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:30.577637 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01d56f54-087c-4d41-b7e7-40944cc40b00 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:30.578250 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01d56f54-087c-4d41-b7e7-40944cc40b00 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:30.578531 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01d56f54-087c-4d41-b7e7-40944cc40b00 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:30.578804 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01d56f54-087c-4d41-b7e7-40944cc40b00 admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:30.579427 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01d56f54-087c-4d41-b7e7-40944cc40b00 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:30.581958 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01d56f54-087c-4d41-b7e7-40944cc40b00 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:30.604707 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1190/2404] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:43:30 2022] POST /identity/v3/users => generated 291 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:43:30.606827 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f2ac242-5123-428f-b8b6-7849365ce194 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:30.617058 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f2ac242-5123-428f-b8b6-7849365ce194 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:30.621675 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71c8ec9a-217d-4a2f-90e9-bb6e6f3e8b69 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:30.622274 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71c8ec9a-217d-4a2f-90e9-bb6e6f3e8b69 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:30.622560 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71c8ec9a-217d-4a2f-90e9-bb6e6f3e8b69 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:30.622835 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71c8ec9a-217d-4a2f-90e9-bb6e6f3e8b69 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:30.623470 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71c8ec9a-217d-4a2f-90e9-bb6e6f3e8b69 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:30.625094 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71c8ec9a-217d-4a2f-90e9-bb6e6f3e8b69 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:30.636389 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1215/2405] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:43:30 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:43:30.640618 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01d56f54-087c-4d41-b7e7-40944cc40b00 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:30.652018 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01d56f54-087c-4d41-b7e7-40944cc40b00 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:30.656900 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dda96e77-6c29-45d3-8ab5-a56e27159a33 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:30.657562 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dda96e77-6c29-45d3-8ab5-a56e27159a33 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:30.657824 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dda96e77-6c29-45d3-8ab5-a56e27159a33 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:30.658094 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dda96e77-6c29-45d3-8ab5-a56e27159a33 admin admin] PATH_INFO: `/v3/projects/b442b49b6aa740aea451de0ba1441790/users/7777a1d47af042a19e9290c3829f4ac4/roles/c51fe7c5ad804040879acfb54269455f` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:30.672826 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dda96e77-6c29-45d3-8ab5-a56e27159a33 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b442b49b6aa740aea451de0ba1441790, user_id=7777a1d47af042a19e9290c3829f4ac4, role_id=c51fe7c5ad804040879acfb54269455f)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:30.675759 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dda96e77-6c29-45d3-8ab5-a56e27159a33 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:30.689268 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1191/2406] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:43:30 2022] PUT /identity/v3/projects/b442b49b6aa740aea451de0ba1441790/users/7777a1d47af042a19e9290c3829f4ac4/roles/c51fe7c5ad804040879acfb54269455f => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:43:30.694031 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71c8ec9a-217d-4a2f-90e9-bb6e6f3e8b69 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:30.700316 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:43:30.704356 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71c8ec9a-217d-4a2f-90e9-bb6e6f3e8b69 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:30.762701 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c54c51c-5f05-4a37-a097-af49532937c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:30.763227 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c54c51c-5f05-4a37-a097-af49532937c3 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:30.763503 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c54c51c-5f05-4a37-a097-af49532937c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:30.763772 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c54c51c-5f05-4a37-a097-af49532937c3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:30.764401 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c54c51c-5f05-4a37-a097-af49532937c3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:30.766084 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c54c51c-5f05-4a37-a097-af49532937c3 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:30.776137 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1216/2407] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:43:30 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:43:30.780342 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dda96e77-6c29-45d3-8ab5-a56e27159a33 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:30.791971 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dda96e77-6c29-45d3-8ab5-a56e27159a33 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:30.796556 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cda7ef5f-a0c6-4664-a306-d49d9f58ffbe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:30.797216 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cda7ef5f-a0c6-4664-a306-d49d9f58ffbe admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:30.797494 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cda7ef5f-a0c6-4664-a306-d49d9f58ffbe admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:30.797765 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cda7ef5f-a0c6-4664-a306-d49d9f58ffbe admin admin] PATH_INFO: `/v3/projects/b442b49b6aa740aea451de0ba1441790/users/7777a1d47af042a19e9290c3829f4ac4/roles/c51fe7c5ad804040879acfb54269455f` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:30.799663 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cda7ef5f-a0c6-4664-a306-d49d9f58ffbe admin admin] RBAC: Authorizing `identity:create_grant(project_id=b442b49b6aa740aea451de0ba1441790, user_id=7777a1d47af042a19e9290c3829f4ac4, role_id=c51fe7c5ad804040879acfb54269455f)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:30.803093 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cda7ef5f-a0c6-4664-a306-d49d9f58ffbe admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:30.814000 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1192/2408] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:43:30 2022] PUT /identity/v3/projects/b442b49b6aa740aea451de0ba1441790/users/7777a1d47af042a19e9290c3829f4ac4/roles/c51fe7c5ad804040879acfb54269455f => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:43:30.825139 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-699aefad-5a3f-46b4-a154-053c0e85200c None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:30.825424 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-699aefad-5a3f-46b4-a154-053c0e85200c None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:30.825693 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-699aefad-5a3f-46b4-a154-053c0e85200c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:30.853331 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-699aefad-5a3f-46b4-a154-053c0e85200c None None] MFA Rules not processed for user `7777a1d47af042a19e9290c3829f4ac4`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:43:30.854566 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-699aefad-5a3f-46b4-a154-053c0e85200c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:43:30.921642 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1217/2409] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:43:30 2022] POST /identity/v3/auth/tokens => generated 3313 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:43:30.933058 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cda7ef5f-a0c6-4664-a306-d49d9f58ffbe admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:30.934779 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-cda7ef5f-a0c6-4664-a306-d49d9f58ffbe admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:43:30.992877 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cda7ef5f-a0c6-4664-a306-d49d9f58ffbe admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:30.996529 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5153d73-b21c-4aa4-ab55-05386a8e8f66 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ffbc136c77b84e8c87128f7247f4ec03', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:30.997239 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5153d73-b21c-4aa4-ab55-05386a8e8f66 service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:30.997584 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5153d73-b21c-4aa4-ab55-05386a8e8f66 service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:30.997883 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5153d73-b21c-4aa4-ab55-05386a8e8f66 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:31.007475 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5153d73-b21c-4aa4-ab55-05386a8e8f66 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:31.008927 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5153d73-b21c-4aa4-ab55-05386a8e8f66 service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:31.018243 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1193/2410] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:43:30 2022] GET /identity/v3/auth/tokens => generated 3313 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:43:32.403034 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-699aefad-5a3f-46b4-a154-053c0e85200c None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:32.465510 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-699aefad-5a3f-46b4-a154-053c0e85200c None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:32.470075 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c903970-9acd-4cae-a242-7615699aaca1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:32.470715 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c903970-9acd-4cae-a242-7615699aaca1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:32.470994 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c903970-9acd-4cae-a242-7615699aaca1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:32.471265 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c903970-9acd-4cae-a242-7615699aaca1 admin admin] PATH_INFO: `/v3/users/faa280fa43674f7c8bd538fd18e9b6f7` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:32.472800 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c903970-9acd-4cae-a242-7615699aaca1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=faa280fa43674f7c8bd538fd18e9b6f7)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:32.474725 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c903970-9acd-4cae-a242-7615699aaca1 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:32.505089 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:43:32.524058 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-6c903970-9acd-4cae-a242-7615699aaca1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'faa280fa43674f7c8bd538fd18e9b6f7'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:32.531513 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-6c903970-9acd-4cae-a242-7615699aaca1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'faa280fa43674f7c8bd538fd18e9b6f7'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:32.544377 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-6c903970-9acd-4cae-a242-7615699aaca1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'faa280fa43674f7c8bd538fd18e9b6f7'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:32.545044 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-6c903970-9acd-4cae-a242-7615699aaca1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'faa280fa43674f7c8bd538fd18e9b6f7'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:32.552568 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-6c903970-9acd-4cae-a242-7615699aaca1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'faa280fa43674f7c8bd538fd18e9b6f7'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:32.558603 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1218/2411] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:43:32 2022] DELETE /identity/v3/users/faa280fa43674f7c8bd538fd18e9b6f7 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:43:32.563040 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-282d9b36-991f-41bb-8e20-99c8102df183 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:32.563524 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-282d9b36-991f-41bb-8e20-99c8102df183 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:32.563637 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-282d9b36-991f-41bb-8e20-99c8102df183 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:32.583108 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-282d9b36-991f-41bb-8e20-99c8102df183 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:43:32.584224 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-282d9b36-991f-41bb-8e20-99c8102df183 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:43:32.588317 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:43:32.643543 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1194/2412] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:43:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:43:32.653559 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c903970-9acd-4cae-a242-7615699aaca1 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:32.654965 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-6c903970-9acd-4cae-a242-7615699aaca1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:43:32.712365 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c903970-9acd-4cae-a242-7615699aaca1 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:32.716325 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6844b7e8-8220-43e3-936f-370083d91854 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:32.716972 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6844b7e8-8220-43e3-936f-370083d91854 service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:32.717258 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6844b7e8-8220-43e3-936f-370083d91854 service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:32.717531 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6844b7e8-8220-43e3-936f-370083d91854 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:32.727173 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6844b7e8-8220-43e3-936f-370083d91854 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:32.728604 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6844b7e8-8220-43e3-936f-370083d91854 service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:32.737927 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1219/2413] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:43:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:43:33.040411 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-282d9b36-991f-41bb-8e20-99c8102df183 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:33.041915 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-282d9b36-991f-41bb-8e20-99c8102df183 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:43:33.057203 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-282d9b36-991f-41bb-8e20-99c8102df183 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:33.062026 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddf6cabf-1cec-48fd-9e86-bdf1ebb707bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:33.062662 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf6cabf-1cec-48fd-9e86-bdf1ebb707bc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:33.062948 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf6cabf-1cec-48fd-9e86-bdf1ebb707bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:33.063224 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf6cabf-1cec-48fd-9e86-bdf1ebb707bc admin admin] PATH_INFO: `/v3/projects/18bad7a155c14488b59ca7a55a7ed0bf` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:33.065059 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddf6cabf-1cec-48fd-9e86-bdf1ebb707bc admin admin] RBAC: Authorizing `identity:delete_project(project_id=18bad7a155c14488b59ca7a55a7ed0bf)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:33.068190 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddf6cabf-1cec-48fd-9e86-bdf1ebb707bc admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:33.115381 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-ddf6cabf-1cec-48fd-9e86-bdf1ebb707bc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '18bad7a155c14488b59ca7a55a7ed0bf', 'request_id': 'req-ddf6cabf-1cec-48fd-9e86-bdf1ebb707bc'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:33.122369 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-ddf6cabf-1cec-48fd-9e86-bdf1ebb707bc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '18bad7a155c14488b59ca7a55a7ed0bf', 'request_id': 'req-ddf6cabf-1cec-48fd-9e86-bdf1ebb707bc'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:33.128075 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-ddf6cabf-1cec-48fd-9e86-bdf1ebb707bc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '18bad7a155c14488b59ca7a55a7ed0bf', 'request_id': 'req-ddf6cabf-1cec-48fd-9e86-bdf1ebb707bc'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:33.139900 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-ddf6cabf-1cec-48fd-9e86-bdf1ebb707bc admin admin] The token cache is being invalidate because project 18bad7a155c14488b59ca7a55a7ed0bf was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:43:33.140188 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-ddf6cabf-1cec-48fd-9e86-bdf1ebb707bc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:33.140846 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-ddf6cabf-1cec-48fd-9e86-bdf1ebb707bc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:33.142330 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1195/2414] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:43:33 2022] DELETE /identity/v3/projects/18bad7a155c14488b59ca7a55a7ed0bf => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:43:33.147155 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6844b7e8-8220-43e3-936f-370083d91854 service neutron] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:33.156883 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6844b7e8-8220-43e3-936f-370083d91854 service neutron] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:33.213161 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e9f0e82-9bb1-4bea-a894-1d080aaf00c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:33.213911 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e9f0e82-9bb1-4bea-a894-1d080aaf00c4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:33.214307 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e9f0e82-9bb1-4bea-a894-1d080aaf00c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:33.214514 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e9f0e82-9bb1-4bea-a894-1d080aaf00c4 admin admin] PATH_INFO: `/v3/users/7777a1d47af042a19e9290c3829f4ac4` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:33.215852 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e9f0e82-9bb1-4bea-a894-1d080aaf00c4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7777a1d47af042a19e9290c3829f4ac4)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:33.218054 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e9f0e82-9bb1-4bea-a894-1d080aaf00c4 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:33.263433 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-1e9f0e82-9bb1-4bea-a894-1d080aaf00c4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7777a1d47af042a19e9290c3829f4ac4'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:33.267195 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:43:33.274482 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-1e9f0e82-9bb1-4bea-a894-1d080aaf00c4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7777a1d47af042a19e9290c3829f4ac4'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:33.289822 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-1e9f0e82-9bb1-4bea-a894-1d080aaf00c4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7777a1d47af042a19e9290c3829f4ac4'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:33.290695 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-1e9f0e82-9bb1-4bea-a894-1d080aaf00c4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7777a1d47af042a19e9290c3829f4ac4'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:33.297846 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-1e9f0e82-9bb1-4bea-a894-1d080aaf00c4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7777a1d47af042a19e9290c3829f4ac4'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:33.303909 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1220/2415] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:43:33 2022] DELETE /identity/v3/users/7777a1d47af042a19e9290c3829f4ac4 => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 04:43:33.576391 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddf6cabf-1cec-48fd-9e86-bdf1ebb707bc admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:33.577897 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-ddf6cabf-1cec-48fd-9e86-bdf1ebb707bc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:43:33.592128 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddf6cabf-1cec-48fd-9e86-bdf1ebb707bc admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:33.650393 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a3a8256-9d48-41ac-9d00-d27b7327d6c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:33.650953 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a3a8256-9d48-41ac-9d00-d27b7327d6c3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:33.651235 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a3a8256-9d48-41ac-9d00-d27b7327d6c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:33.651512 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a3a8256-9d48-41ac-9d00-d27b7327d6c3 admin admin] PATH_INFO: `/v3/projects/b442b49b6aa740aea451de0ba1441790` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:33.652660 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a3a8256-9d48-41ac-9d00-d27b7327d6c3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b442b49b6aa740aea451de0ba1441790)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:33.654580 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a3a8256-9d48-41ac-9d00-d27b7327d6c3 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:33.692091 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:43:33.704144 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-0a3a8256-9d48-41ac-9d00-d27b7327d6c3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b442b49b6aa740aea451de0ba1441790', 'request_id': 'req-0a3a8256-9d48-41ac-9d00-d27b7327d6c3'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:33.711503 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-0a3a8256-9d48-41ac-9d00-d27b7327d6c3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b442b49b6aa740aea451de0ba1441790', 'request_id': 'req-0a3a8256-9d48-41ac-9d00-d27b7327d6c3'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:33.717207 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-0a3a8256-9d48-41ac-9d00-d27b7327d6c3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b442b49b6aa740aea451de0ba1441790', 'request_id': 'req-0a3a8256-9d48-41ac-9d00-d27b7327d6c3'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:33.728871 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-0a3a8256-9d48-41ac-9d00-d27b7327d6c3 admin admin] The token cache is being invalidate because project b442b49b6aa740aea451de0ba1441790 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:43:33.729161 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-0a3a8256-9d48-41ac-9d00-d27b7327d6c3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:33.729800 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-0a3a8256-9d48-41ac-9d00-d27b7327d6c3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:33.731272 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1196/2416] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:43:33 2022] DELETE /identity/v3/projects/b442b49b6aa740aea451de0ba1441790 => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:43:33.736338 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d40006-f252-4cbe-80e5-8e681defd154 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:33.736842 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d40006-f252-4cbe-80e5-8e681defd154 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:33.736938 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d40006-f252-4cbe-80e5-8e681defd154 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:33.757309 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-39d40006-f252-4cbe-80e5-8e681defd154 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:43:33.758524 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-39d40006-f252-4cbe-80e5-8e681defd154 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:43:33.820667 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1221/2417] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:43:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 04:43:33.828230 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be761fe-253f-43c6-bd29-ed501a09d420 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:33.828515 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be761fe-253f-43c6-bd29-ed501a09d420 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:33.828826 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be761fe-253f-43c6-bd29-ed501a09d420 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:33.849946 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-5be761fe-253f-43c6-bd29-ed501a09d420 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:43:33.851159 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-5be761fe-253f-43c6-bd29-ed501a09d420 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:43:33.859129 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1197/2418] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:43:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:43:33.863312 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39d40006-f252-4cbe-80e5-8e681defd154 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:33.873416 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39d40006-f252-4cbe-80e5-8e681defd154 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:33.878103 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77d55889-8dad-4427-ad3f-239febae27e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:33.878706 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77d55889-8dad-4427-ad3f-239febae27e6 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:33.878989 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77d55889-8dad-4427-ad3f-239febae27e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:33.879309 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77d55889-8dad-4427-ad3f-239febae27e6 admin admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:33.879956 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77d55889-8dad-4427-ad3f-239febae27e6 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:43:33.880263 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77d55889-8dad-4427-ad3f-239febae27e6 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:33.881949 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77d55889-8dad-4427-ad3f-239febae27e6 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:33.893345 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1222/2419] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Sep 25 04:43:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:43:33.898307 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5be761fe-253f-43c6-bd29-ed501a09d420 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:33.907667 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5be761fe-253f-43c6-bd29-ed501a09d420 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:33.913157 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae473c5d-4c37-41cf-ad5c-95dbe8d4b94d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:33.913754 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae473c5d-4c37-41cf-ad5c-95dbe8d4b94d admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:33.914037 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae473c5d-4c37-41cf-ad5c-95dbe8d4b94d admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:33.914310 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae473c5d-4c37-41cf-ad5c-95dbe8d4b94d admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:33.914938 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae473c5d-4c37-41cf-ad5c-95dbe8d4b94d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:33.916866 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae473c5d-4c37-41cf-ad5c-95dbe8d4b94d admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:33.946905 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1198/2420] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:43:33 2022] POST /identity/v3/projects => generated 369 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:43:33.948062 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77d55889-8dad-4427-ad3f-239febae27e6 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:33.958853 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77d55889-8dad-4427-ad3f-239febae27e6 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:34.017001 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db299973-86b3-48d0-bff4-3104dad97910 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:34.017613 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db299973-86b3-48d0-bff4-3104dad97910 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:34.017893 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db299973-86b3-48d0-bff4-3104dad97910 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:34.018168 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db299973-86b3-48d0-bff4-3104dad97910 admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:34.018800 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db299973-86b3-48d0-bff4-3104dad97910 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:34.021457 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db299973-86b3-48d0-bff4-3104dad97910 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:34.045587 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1223/2421] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:43:33 2022] POST /identity/v3/users => generated 285 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 04:43:34.049659 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae473c5d-4c37-41cf-ad5c-95dbe8d4b94d admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:34.061367 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae473c5d-4c37-41cf-ad5c-95dbe8d4b94d admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:34.067495 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab60ced0-7a3c-4aed-9d01-db4ee4361f37 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:34.068820 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab60ced0-7a3c-4aed-9d01-db4ee4361f37 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:34.069114 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab60ced0-7a3c-4aed-9d01-db4ee4361f37 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:34.069390 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab60ced0-7a3c-4aed-9d01-db4ee4361f37 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:34.070042 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab60ced0-7a3c-4aed-9d01-db4ee4361f37 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:34.071708 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab60ced0-7a3c-4aed-9d01-db4ee4361f37 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:34.082068 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1199/2422] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:43:34 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:43:34.086623 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db299973-86b3-48d0-bff4-3104dad97910 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:34.097190 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db299973-86b3-48d0-bff4-3104dad97910 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:34.102678 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68fa747d-0271-40ec-8fe1-e7784c071b11 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:34.103294 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68fa747d-0271-40ec-8fe1-e7784c071b11 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:34.103575 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68fa747d-0271-40ec-8fe1-e7784c071b11 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:34.103850 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68fa747d-0271-40ec-8fe1-e7784c071b11 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:34.104492 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68fa747d-0271-40ec-8fe1-e7784c071b11 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:34.106119 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68fa747d-0271-40ec-8fe1-e7784c071b11 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:34.117354 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1224/2423] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:43:34 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:43:34.121577 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab60ced0-7a3c-4aed-9d01-db4ee4361f37 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:34.132871 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab60ced0-7a3c-4aed-9d01-db4ee4361f37 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:34.137713 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da52c892-0fd8-40ac-a782-a43e454edec8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:34.138377 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da52c892-0fd8-40ac-a782-a43e454edec8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:34.138678 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da52c892-0fd8-40ac-a782-a43e454edec8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:34.138958 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da52c892-0fd8-40ac-a782-a43e454edec8 admin admin] PATH_INFO: `/v3/projects/fbe4c138eaf54fddae63ae1bf4728fa7/users/29efbd0d963f4d5fa3431565cef0e52a/roles/8f6cc574056c4278bc9778f14dc4b411` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:34.154423 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da52c892-0fd8-40ac-a782-a43e454edec8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fbe4c138eaf54fddae63ae1bf4728fa7, user_id=29efbd0d963f4d5fa3431565cef0e52a, role_id=8f6cc574056c4278bc9778f14dc4b411)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:34.157131 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da52c892-0fd8-40ac-a782-a43e454edec8 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:34.172170 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1200/2424] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:43:34 2022] PUT /identity/v3/projects/fbe4c138eaf54fddae63ae1bf4728fa7/users/29efbd0d963f4d5fa3431565cef0e52a/roles/8f6cc574056c4278bc9778f14dc4b411 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:43:34.186763 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1551af99-0dc9-402e-93aa-0289f28d7768 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:34.187119 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1551af99-0dc9-402e-93aa-0289f28d7768 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:34.187588 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1551af99-0dc9-402e-93aa-0289f28d7768 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:34.217066 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-1551af99-0dc9-402e-93aa-0289f28d7768 None None] MFA Rules not processed for user `29efbd0d963f4d5fa3431565cef0e52a`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:43:34.218334 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-1551af99-0dc9-402e-93aa-0289f28d7768 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:43:34.239216 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:DELETE=3,keystone:SELECT=94,keystone:INSERT=3 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:43:34.288257 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1225/2425] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:43:34 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:43:34.292772 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da52c892-0fd8-40ac-a782-a43e454edec8 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:34.355948 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da52c892-0fd8-40ac-a782-a43e454edec8 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:34.360824 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a027bc24-0914-4b4b-a7ff-e06ca0526adb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:34.361433 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a027bc24-0914-4b4b-a7ff-e06ca0526adb admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:34.361716 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a027bc24-0914-4b4b-a7ff-e06ca0526adb admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:34.362001 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a027bc24-0914-4b4b-a7ff-e06ca0526adb admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:34.362621 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a027bc24-0914-4b4b-a7ff-e06ca0526adb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:34.364597 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a027bc24-0914-4b4b-a7ff-e06ca0526adb admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:34.392395 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1201/2426] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:43:34 2022] POST /identity/v3/projects => generated 365 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:43:34.397132 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1551af99-0dc9-402e-93aa-0289f28d7768 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:34.407872 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1551af99-0dc9-402e-93aa-0289f28d7768 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:34.471379 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3687a0dc-f29d-49b0-8f66-7e5d347a89aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:34.472017 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3687a0dc-f29d-49b0-8f66-7e5d347a89aa admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:34.472302 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3687a0dc-f29d-49b0-8f66-7e5d347a89aa admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:34.472579 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3687a0dc-f29d-49b0-8f66-7e5d347a89aa admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:34.473839 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3687a0dc-f29d-49b0-8f66-7e5d347a89aa admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:34.475840 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3687a0dc-f29d-49b0-8f66-7e5d347a89aa admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:34.502267 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1226/2427] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:43:34 2022] POST /identity/v3/users => generated 289 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 04:43:34.503619 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a027bc24-0914-4b4b-a7ff-e06ca0526adb admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:34.514821 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a027bc24-0914-4b4b-a7ff-e06ca0526adb admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:34.522299 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76d5b757-6daf-48cb-bbd7-32f50d5e1814 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:34.522903 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76d5b757-6daf-48cb-bbd7-32f50d5e1814 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:34.523188 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76d5b757-6daf-48cb-bbd7-32f50d5e1814 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:34.523458 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76d5b757-6daf-48cb-bbd7-32f50d5e1814 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:34.524096 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76d5b757-6daf-48cb-bbd7-32f50d5e1814 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:34.525728 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76d5b757-6daf-48cb-bbd7-32f50d5e1814 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:34.540222 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1202/2428] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:43:34 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:43:34.541422 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3687a0dc-f29d-49b0-8f66-7e5d347a89aa admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:34.551612 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3687a0dc-f29d-49b0-8f66-7e5d347a89aa admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:34.556502 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50310302-311c-490c-aa15-e2f926840038 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:34.557150 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50310302-311c-490c-aa15-e2f926840038 admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:34.557432 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50310302-311c-490c-aa15-e2f926840038 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:34.557708 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50310302-311c-490c-aa15-e2f926840038 admin admin] PATH_INFO: `/v3/projects/342d747f53004ba6b5b8d42815c8f472/users/33eaba0ffc85454085069ac6e7ffe86f/roles/c51fe7c5ad804040879acfb54269455f` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:34.572784 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50310302-311c-490c-aa15-e2f926840038 admin admin] RBAC: Authorizing `identity:create_grant(project_id=342d747f53004ba6b5b8d42815c8f472, user_id=33eaba0ffc85454085069ac6e7ffe86f, role_id=c51fe7c5ad804040879acfb54269455f)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:34.575434 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50310302-311c-490c-aa15-e2f926840038 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:34.590858 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1227/2429] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:43:34 2022] PUT /identity/v3/projects/342d747f53004ba6b5b8d42815c8f472/users/33eaba0ffc85454085069ac6e7ffe86f/roles/c51fe7c5ad804040879acfb54269455f => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:43:34.594738 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76d5b757-6daf-48cb-bbd7-32f50d5e1814 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:34.606175 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76d5b757-6daf-48cb-bbd7-32f50d5e1814 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:34.638101 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:DELETE=5,keystone:SELECT=91,keystone:INSERT=4 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:43:34.668570 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-083af681-bbb9-41e5-88e9-465b3cdbfd48 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:34.669216 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-083af681-bbb9-41e5-88e9-465b3cdbfd48 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:34.669502 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-083af681-bbb9-41e5-88e9-465b3cdbfd48 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:34.669778 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-083af681-bbb9-41e5-88e9-465b3cdbfd48 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:34.670419 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-083af681-bbb9-41e5-88e9-465b3cdbfd48 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:34.672030 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-083af681-bbb9-41e5-88e9-465b3cdbfd48 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:34.685427 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1203/2430] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:43:34 2022] GET /identity/v3/roles => generated 1420 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:43:34.686688 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50310302-311c-490c-aa15-e2f926840038 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:34.697545 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50310302-311c-490c-aa15-e2f926840038 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:34.702881 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8df950e-370f-4170-a826-1c8a71003c46 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:34.703544 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8df950e-370f-4170-a826-1c8a71003c46 admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:34.703826 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8df950e-370f-4170-a826-1c8a71003c46 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:34.704104 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8df950e-370f-4170-a826-1c8a71003c46 admin admin] PATH_INFO: `/v3/projects/342d747f53004ba6b5b8d42815c8f472/users/33eaba0ffc85454085069ac6e7ffe86f/roles/c51fe7c5ad804040879acfb54269455f` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:34.706060 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8df950e-370f-4170-a826-1c8a71003c46 admin admin] RBAC: Authorizing `identity:create_grant(project_id=342d747f53004ba6b5b8d42815c8f472, user_id=33eaba0ffc85454085069ac6e7ffe86f, role_id=c51fe7c5ad804040879acfb54269455f)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:34.709371 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8df950e-370f-4170-a826-1c8a71003c46 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:34.720219 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1228/2431] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:43:34 2022] PUT /identity/v3/projects/342d747f53004ba6b5b8d42815c8f472/users/33eaba0ffc85454085069ac6e7ffe86f/roles/c51fe7c5ad804040879acfb54269455f => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:43:34.736128 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56d97f32-5cb4-40cd-912a-87622a1379e6 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:34.736416 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56d97f32-5cb4-40cd-912a-87622a1379e6 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:34.736719 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56d97f32-5cb4-40cd-912a-87622a1379e6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:34.764522 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-56d97f32-5cb4-40cd-912a-87622a1379e6 None None] MFA Rules not processed for user `33eaba0ffc85454085069ac6e7ffe86f`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:43:34.765852 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-56d97f32-5cb4-40cd-912a-87622a1379e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:43:34.838792 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1204/2432] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:43:34 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:43:34.850897 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8df950e-370f-4170-a826-1c8a71003c46 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:34.853206 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-d8df950e-370f-4170-a826-1c8a71003c46 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:43:34.912932 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8df950e-370f-4170-a826-1c8a71003c46 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:34.916532 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d966830a-14dd-4739-850d-0c0d33c3ae55 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ffbc136c77b84e8c87128f7247f4ec03', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:34.917172 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d966830a-14dd-4739-850d-0c0d33c3ae55 service cinder] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:34.917446 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d966830a-14dd-4739-850d-0c0d33c3ae55 service cinder] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:34.917718 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d966830a-14dd-4739-850d-0c0d33c3ae55 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:34.926863 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d966830a-14dd-4739-850d-0c0d33c3ae55 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:34.928274 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d966830a-14dd-4739-850d-0c0d33c3ae55 service cinder] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:34.939097 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1229/2433] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:43:34 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:43:34.986165 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56d97f32-5cb4-40cd-912a-87622a1379e6 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:34.987582 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-56d97f32-5cb4-40cd-912a-87622a1379e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:43:35.000551 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56d97f32-5cb4-40cd-912a-87622a1379e6 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:35.004952 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b841de2a-e3d3-44d7-babc-4673917ae166 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ffbc136c77b84e8c87128f7247f4ec03', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:35.005568 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b841de2a-e3d3-44d7-babc-4673917ae166 service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:35.005848 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b841de2a-e3d3-44d7-babc-4673917ae166 service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:35.006245 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b841de2a-e3d3-44d7-babc-4673917ae166 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:35.065154 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b841de2a-e3d3-44d7-babc-4673917ae166 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:35.066604 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b841de2a-e3d3-44d7-babc-4673917ae166 service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:35.076528 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1205/2434] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:43:34 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:43:38.420655 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d966830a-14dd-4739-850d-0c0d33c3ae55 service cinder] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:38.442341 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:43:38.487915 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d966830a-14dd-4739-850d-0c0d33c3ae55 service cinder] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:38.492540 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5017ed36-c6ae-4dbd-b86b-cfb934c4093d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:38.493200 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5017ed36-c6ae-4dbd-b86b-cfb934c4093d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:38.493477 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5017ed36-c6ae-4dbd-b86b-cfb934c4093d admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:38.493746 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5017ed36-c6ae-4dbd-b86b-cfb934c4093d admin admin] PATH_INFO: `/v3/users/29efbd0d963f4d5fa3431565cef0e52a` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:38.494881 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5017ed36-c6ae-4dbd-b86b-cfb934c4093d admin admin] RBAC: Authorizing `identity:delete_user(user_id=29efbd0d963f4d5fa3431565cef0e52a)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:38.496819 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5017ed36-c6ae-4dbd-b86b-cfb934c4093d admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:38.541668 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-5017ed36-c6ae-4dbd-b86b-cfb934c4093d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '29efbd0d963f4d5fa3431565cef0e52a'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:38.548443 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-5017ed36-c6ae-4dbd-b86b-cfb934c4093d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '29efbd0d963f4d5fa3431565cef0e52a'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:38.560922 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-5017ed36-c6ae-4dbd-b86b-cfb934c4093d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '29efbd0d963f4d5fa3431565cef0e52a'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:38.561559 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-5017ed36-c6ae-4dbd-b86b-cfb934c4093d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '29efbd0d963f4d5fa3431565cef0e52a'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:38.569144 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-5017ed36-c6ae-4dbd-b86b-cfb934c4093d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '29efbd0d963f4d5fa3431565cef0e52a'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:38.574864 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1230/2435] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:43:38 2022] DELETE /identity/v3/users/29efbd0d963f4d5fa3431565cef0e52a => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:43:38.579324 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5284623e-4758-4c47-aa93-5b23fd5f9a8f None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:38.579797 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5284623e-4758-4c47-aa93-5b23fd5f9a8f None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:38.579902 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5284623e-4758-4c47-aa93-5b23fd5f9a8f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:38.599327 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-5284623e-4758-4c47-aa93-5b23fd5f9a8f None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:43:38.600536 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-5284623e-4758-4c47-aa93-5b23fd5f9a8f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:43:38.646482 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:43:38.658692 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1206/2436] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:43:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:43:38.668631 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5017ed36-c6ae-4dbd-b86b-cfb934c4093d admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:38.670054 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-5017ed36-c6ae-4dbd-b86b-cfb934c4093d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:43:38.727048 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5017ed36-c6ae-4dbd-b86b-cfb934c4093d admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:38.730752 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-626cd0cb-3ecd-4706-8f04-5d2021a1fe60 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:38.731346 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-626cd0cb-3ecd-4706-8f04-5d2021a1fe60 service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:38.731626 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-626cd0cb-3ecd-4706-8f04-5d2021a1fe60 service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:38.731901 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-626cd0cb-3ecd-4706-8f04-5d2021a1fe60 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:38.741341 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-626cd0cb-3ecd-4706-8f04-5d2021a1fe60 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:38.742762 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-626cd0cb-3ecd-4706-8f04-5d2021a1fe60 service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:38.752097 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1231/2437] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:43:38 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:43:39.026268 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5284623e-4758-4c47-aa93-5b23fd5f9a8f None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:39.027879 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-5284623e-4758-4c47-aa93-5b23fd5f9a8f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:43:39.041204 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5284623e-4758-4c47-aa93-5b23fd5f9a8f None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:39.046193 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ccad7c1-27a3-499f-a69b-1efdb0025706 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:39.046842 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ccad7c1-27a3-499f-a69b-1efdb0025706 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:39.047129 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ccad7c1-27a3-499f-a69b-1efdb0025706 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:39.047408 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ccad7c1-27a3-499f-a69b-1efdb0025706 admin admin] PATH_INFO: `/v3/projects/fbe4c138eaf54fddae63ae1bf4728fa7` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:39.048599 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ccad7c1-27a3-499f-a69b-1efdb0025706 admin admin] RBAC: Authorizing `identity:delete_project(project_id=fbe4c138eaf54fddae63ae1bf4728fa7)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:39.050581 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ccad7c1-27a3-499f-a69b-1efdb0025706 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:39.099260 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-1ccad7c1-27a3-499f-a69b-1efdb0025706 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fbe4c138eaf54fddae63ae1bf4728fa7', 'request_id': 'req-1ccad7c1-27a3-499f-a69b-1efdb0025706'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:39.106703 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-1ccad7c1-27a3-499f-a69b-1efdb0025706 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fbe4c138eaf54fddae63ae1bf4728fa7', 'request_id': 'req-1ccad7c1-27a3-499f-a69b-1efdb0025706'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:39.112655 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-1ccad7c1-27a3-499f-a69b-1efdb0025706 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fbe4c138eaf54fddae63ae1bf4728fa7', 'request_id': 'req-1ccad7c1-27a3-499f-a69b-1efdb0025706'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:39.123660 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-1ccad7c1-27a3-499f-a69b-1efdb0025706 admin admin] The token cache is being invalidate because project fbe4c138eaf54fddae63ae1bf4728fa7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:43:39.123953 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-1ccad7c1-27a3-499f-a69b-1efdb0025706 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:39.124594 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-1ccad7c1-27a3-499f-a69b-1efdb0025706 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:39.126101 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1207/2438] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:43:39 2022] DELETE /identity/v3/projects/fbe4c138eaf54fddae63ae1bf4728fa7 => generated 0 bytes in 100 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:43:39.130146 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-626cd0cb-3ecd-4706-8f04-5d2021a1fe60 service neutron] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:39.139676 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-626cd0cb-3ecd-4706-8f04-5d2021a1fe60 service neutron] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:39.178482 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:43:39.198727 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-959cbbb2-5e8e-4d39-aa37-ee1db693f3bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:39.199365 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-959cbbb2-5e8e-4d39-aa37-ee1db693f3bf admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:39.199648 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-959cbbb2-5e8e-4d39-aa37-ee1db693f3bf admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:39.199926 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-959cbbb2-5e8e-4d39-aa37-ee1db693f3bf admin admin] PATH_INFO: `/v3/users/33eaba0ffc85454085069ac6e7ffe86f` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:39.201150 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-959cbbb2-5e8e-4d39-aa37-ee1db693f3bf admin admin] RBAC: Authorizing `identity:delete_user(user_id=33eaba0ffc85454085069ac6e7ffe86f)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:39.203032 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-959cbbb2-5e8e-4d39-aa37-ee1db693f3bf admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:39.251248 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-959cbbb2-5e8e-4d39-aa37-ee1db693f3bf admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '33eaba0ffc85454085069ac6e7ffe86f'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:39.259167 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-959cbbb2-5e8e-4d39-aa37-ee1db693f3bf admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '33eaba0ffc85454085069ac6e7ffe86f'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:39.271451 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-959cbbb2-5e8e-4d39-aa37-ee1db693f3bf admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '33eaba0ffc85454085069ac6e7ffe86f'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:39.272362 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-959cbbb2-5e8e-4d39-aa37-ee1db693f3bf admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '33eaba0ffc85454085069ac6e7ffe86f'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:39.279443 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-959cbbb2-5e8e-4d39-aa37-ee1db693f3bf admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '33eaba0ffc85454085069ac6e7ffe86f'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:39.285211 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1232/2439] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:43:39 2022] DELETE /identity/v3/users/33eaba0ffc85454085069ac6e7ffe86f => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 04:43:39.557168 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ccad7c1-27a3-499f-a69b-1efdb0025706 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:39.558654 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-1ccad7c1-27a3-499f-a69b-1efdb0025706 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:43:39.580414 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ccad7c1-27a3-499f-a69b-1efdb0025706 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:39.635941 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98ba2b3e-d5f1-4551-aa9c-c44d9735a152 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:39.636590 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98ba2b3e-d5f1-4551-aa9c-c44d9735a152 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:39.636903 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98ba2b3e-d5f1-4551-aa9c-c44d9735a152 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:39.637180 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98ba2b3e-d5f1-4551-aa9c-c44d9735a152 admin admin] PATH_INFO: `/v3/projects/342d747f53004ba6b5b8d42815c8f472` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:39.638316 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98ba2b3e-d5f1-4551-aa9c-c44d9735a152 admin admin] RBAC: Authorizing `identity:delete_project(project_id=342d747f53004ba6b5b8d42815c8f472)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:39.640283 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98ba2b3e-d5f1-4551-aa9c-c44d9735a152 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:39.686995 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-98ba2b3e-d5f1-4551-aa9c-c44d9735a152 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '342d747f53004ba6b5b8d42815c8f472', 'request_id': 'req-98ba2b3e-d5f1-4551-aa9c-c44d9735a152'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:39.694768 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-98ba2b3e-d5f1-4551-aa9c-c44d9735a152 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '342d747f53004ba6b5b8d42815c8f472', 'request_id': 'req-98ba2b3e-d5f1-4551-aa9c-c44d9735a152'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:39.700619 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-98ba2b3e-d5f1-4551-aa9c-c44d9735a152 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '342d747f53004ba6b5b8d42815c8f472', 'request_id': 'req-98ba2b3e-d5f1-4551-aa9c-c44d9735a152'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:39.712314 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-98ba2b3e-d5f1-4551-aa9c-c44d9735a152 admin admin] The token cache is being invalidate because project 342d747f53004ba6b5b8d42815c8f472 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:43:39.712602 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-98ba2b3e-d5f1-4551-aa9c-c44d9735a152 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:39.713518 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-98ba2b3e-d5f1-4551-aa9c-c44d9735a152 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:39.715012 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1208/2440] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:43:39 2022] DELETE /identity/v3/projects/342d747f53004ba6b5b8d42815c8f472 => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:43:39.719940 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65b9c5ec-f6d8-4693-b331-2a718b407fa7 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:39.720450 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65b9c5ec-f6d8-4693-b331-2a718b407fa7 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:39.720552 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65b9c5ec-f6d8-4693-b331-2a718b407fa7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:39.740347 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-65b9c5ec-f6d8-4693-b331-2a718b407fa7 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:43:39.741511 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-65b9c5ec-f6d8-4693-b331-2a718b407fa7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:43:39.799543 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1233/2441] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:43:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:43:39.809106 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c7c4c0-8e61-45df-bac3-9c0a60912862 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:39.809396 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c7c4c0-8e61-45df-bac3-9c0a60912862 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:39.809676 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c7c4c0-8e61-45df-bac3-9c0a60912862 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:39.824775 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:43:39.831211 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-95c7c4c0-8e61-45df-bac3-9c0a60912862 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:43:39.833198 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-95c7c4c0-8e61-45df-bac3-9c0a60912862 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:43:39.841568 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1209/2442] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:43:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:43:39.845714 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65b9c5ec-f6d8-4693-b331-2a718b407fa7 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:39.855374 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65b9c5ec-f6d8-4693-b331-2a718b407fa7 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:39.859959 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28f1fabc-f648-4a45-afa4-fe527bfd924b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:39.860558 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28f1fabc-f648-4a45-afa4-fe527bfd924b admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:39.860871 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28f1fabc-f648-4a45-afa4-fe527bfd924b admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:39.861144 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28f1fabc-f648-4a45-afa4-fe527bfd924b admin admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:39.861781 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28f1fabc-f648-4a45-afa4-fe527bfd924b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:43:39.862089 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28f1fabc-f648-4a45-afa4-fe527bfd924b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:39.864179 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28f1fabc-f648-4a45-afa4-fe527bfd924b admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:39.875968 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1234/2443] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Sep 25 04:43:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:43:39.880849 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95c7c4c0-8e61-45df-bac3-9c0a60912862 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:39.890447 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95c7c4c0-8e61-45df-bac3-9c0a60912862 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:39.895171 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1611404-2e7b-4cbc-a904-4940d773555e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:39.895791 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1611404-2e7b-4cbc-a904-4940d773555e admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:39.896072 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1611404-2e7b-4cbc-a904-4940d773555e admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:39.896347 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1611404-2e7b-4cbc-a904-4940d773555e admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:39.897056 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1611404-2e7b-4cbc-a904-4940d773555e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:39.898966 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1611404-2e7b-4cbc-a904-4940d773555e admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:39.926528 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1210/2444] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:43:39 2022] POST /identity/v3/projects => generated 385 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:43:39.931122 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28f1fabc-f648-4a45-afa4-fe527bfd924b admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:39.941181 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28f1fabc-f648-4a45-afa4-fe527bfd924b admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:39.984961 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:43:40.000222 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40481710-1ce0-4dc8-be5a-0df010711e23 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:40.000798 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40481710-1ce0-4dc8-be5a-0df010711e23 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:40.001089 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40481710-1ce0-4dc8-be5a-0df010711e23 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:40.001360 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40481710-1ce0-4dc8-be5a-0df010711e23 admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:40.002004 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40481710-1ce0-4dc8-be5a-0df010711e23 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:40.003932 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40481710-1ce0-4dc8-be5a-0df010711e23 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:40.028409 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1235/2445] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:43:39 2022] POST /identity/v3/users => generated 293 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:43:40.029412 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1611404-2e7b-4cbc-a904-4940d773555e admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:40.039822 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1611404-2e7b-4cbc-a904-4940d773555e admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:40.045007 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c13b8681-a054-44b4-a14d-f637abf054b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:40.045611 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c13b8681-a054-44b4-a14d-f637abf054b7 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:40.045896 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c13b8681-a054-44b4-a14d-f637abf054b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:40.046199 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c13b8681-a054-44b4-a14d-f637abf054b7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:40.046857 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c13b8681-a054-44b4-a14d-f637abf054b7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:40.048458 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c13b8681-a054-44b4-a14d-f637abf054b7 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:40.059158 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1211/2446] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:43:40 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 25 04:43:40.063121 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40481710-1ce0-4dc8-be5a-0df010711e23 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:40.073684 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40481710-1ce0-4dc8-be5a-0df010711e23 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:40.078524 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6824ec0-d763-4118-aa8d-0cb07ccd0ed4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:40.079122 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6824ec0-d763-4118-aa8d-0cb07ccd0ed4 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:40.079405 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6824ec0-d763-4118-aa8d-0cb07ccd0ed4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:40.079685 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6824ec0-d763-4118-aa8d-0cb07ccd0ed4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:40.080322 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6824ec0-d763-4118-aa8d-0cb07ccd0ed4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:40.081965 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6824ec0-d763-4118-aa8d-0cb07ccd0ed4 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:40.092788 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1236/2447] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:43:40 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:43:40.097088 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c13b8681-a054-44b4-a14d-f637abf054b7 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:40.107254 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c13b8681-a054-44b4-a14d-f637abf054b7 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:40.111926 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79214d38-5092-4fad-aaf1-fd13f5fda96a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:40.112572 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79214d38-5092-4fad-aaf1-fd13f5fda96a admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:40.112889 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79214d38-5092-4fad-aaf1-fd13f5fda96a admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:40.113168 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79214d38-5092-4fad-aaf1-fd13f5fda96a admin admin] PATH_INFO: `/v3/projects/9cac1a31801a470f83d857e094caed5a/users/33ee94fa37f04587a15460d867d9cb56/roles/8f6cc574056c4278bc9778f14dc4b411` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:40.127910 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79214d38-5092-4fad-aaf1-fd13f5fda96a admin admin] RBAC: Authorizing `identity:create_grant(project_id=9cac1a31801a470f83d857e094caed5a, user_id=33ee94fa37f04587a15460d867d9cb56, role_id=8f6cc574056c4278bc9778f14dc4b411)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:40.131315 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79214d38-5092-4fad-aaf1-fd13f5fda96a admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:40.144015 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1212/2448] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:43:40 2022] PUT /identity/v3/projects/9cac1a31801a470f83d857e094caed5a/users/33ee94fa37f04587a15460d867d9cb56/roles/8f6cc574056c4278bc9778f14dc4b411 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:43:40.154439 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-596ca7a2-0922-4b39-b72f-abf86a2a05fb None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:40.154737 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-596ca7a2-0922-4b39-b72f-abf86a2a05fb None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:40.155040 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-596ca7a2-0922-4b39-b72f-abf86a2a05fb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:40.183193 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-596ca7a2-0922-4b39-b72f-abf86a2a05fb None None] MFA Rules not processed for user `33ee94fa37f04587a15460d867d9cb56`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:43:40.184399 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-596ca7a2-0922-4b39-b72f-abf86a2a05fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:43:40.248645 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1237/2449] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:43:40 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:43:40.253349 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79214d38-5092-4fad-aaf1-fd13f5fda96a admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:40.314979 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79214d38-5092-4fad-aaf1-fd13f5fda96a admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:40.319827 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10896e73-cc12-45e5-aaff-bdfa65537179 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:40.320425 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10896e73-cc12-45e5-aaff-bdfa65537179 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:40.320740 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10896e73-cc12-45e5-aaff-bdfa65537179 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:40.321018 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10896e73-cc12-45e5-aaff-bdfa65537179 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:40.321636 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10896e73-cc12-45e5-aaff-bdfa65537179 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:40.323557 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10896e73-cc12-45e5-aaff-bdfa65537179 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:40.352462 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1213/2450] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:43:40 2022] POST /identity/v3/projects => generated 383 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:43:40.356500 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-596ca7a2-0922-4b39-b72f-abf86a2a05fb None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:40.368334 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-596ca7a2-0922-4b39-b72f-abf86a2a05fb None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:40.423668 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05ecb43a-b44f-4a03-9d1f-335840491cd8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:40.424274 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05ecb43a-b44f-4a03-9d1f-335840491cd8 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:40.424552 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05ecb43a-b44f-4a03-9d1f-335840491cd8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:40.424854 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05ecb43a-b44f-4a03-9d1f-335840491cd8 admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:40.425484 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05ecb43a-b44f-4a03-9d1f-335840491cd8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:40.428253 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05ecb43a-b44f-4a03-9d1f-335840491cd8 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:40.449055 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1238/2451] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:43:40 2022] POST /identity/v3/users => generated 298 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:43:40.453241 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10896e73-cc12-45e5-aaff-bdfa65537179 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:40.464080 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10896e73-cc12-45e5-aaff-bdfa65537179 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:40.468905 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca604502-286c-46e6-afac-0cf4154c5ecf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:40.469508 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca604502-286c-46e6-afac-0cf4154c5ecf admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:40.469789 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca604502-286c-46e6-afac-0cf4154c5ecf admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:40.470065 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca604502-286c-46e6-afac-0cf4154c5ecf admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:40.470710 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca604502-286c-46e6-afac-0cf4154c5ecf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:40.472333 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca604502-286c-46e6-afac-0cf4154c5ecf admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:40.483975 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1214/2452] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:43:40 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:43:40.488136 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05ecb43a-b44f-4a03-9d1f-335840491cd8 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:40.499017 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05ecb43a-b44f-4a03-9d1f-335840491cd8 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:40.503790 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c7853b3-f869-46e3-be54-ed5cd90134fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:40.504411 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c7853b3-f869-46e3-be54-ed5cd90134fa admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:40.504727 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c7853b3-f869-46e3-be54-ed5cd90134fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:40.505007 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c7853b3-f869-46e3-be54-ed5cd90134fa admin admin] PATH_INFO: `/v3/projects/a68704962f7346998beb980b920bbbb2/users/5121e8ae28b743dbaef18f5ed9a82b33/roles/c51fe7c5ad804040879acfb54269455f` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:40.519373 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c7853b3-f869-46e3-be54-ed5cd90134fa admin admin] RBAC: Authorizing `identity:create_grant(project_id=a68704962f7346998beb980b920bbbb2, user_id=5121e8ae28b743dbaef18f5ed9a82b33, role_id=c51fe7c5ad804040879acfb54269455f)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:40.522896 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c7853b3-f869-46e3-be54-ed5cd90134fa admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:40.536134 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1239/2453] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:43:40 2022] PUT /identity/v3/projects/a68704962f7346998beb980b920bbbb2/users/5121e8ae28b743dbaef18f5ed9a82b33/roles/c51fe7c5ad804040879acfb54269455f => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:43:40.540179 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca604502-286c-46e6-afac-0cf4154c5ecf admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:40.550239 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca604502-286c-46e6-afac-0cf4154c5ecf admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:40.606906 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06e87e93-f6f5-4c05-9c7c-c47ea09d4fbe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:40.607530 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06e87e93-f6f5-4c05-9c7c-c47ea09d4fbe admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:40.607753 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06e87e93-f6f5-4c05-9c7c-c47ea09d4fbe admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:40.608022 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06e87e93-f6f5-4c05-9c7c-c47ea09d4fbe admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:40.608702 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06e87e93-f6f5-4c05-9c7c-c47ea09d4fbe admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:40.610292 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06e87e93-f6f5-4c05-9c7c-c47ea09d4fbe admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:40.620169 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1215/2454] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:43:40 2022] GET /identity/v3/roles => generated 1420 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:43:40.624274 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c7853b3-f869-46e3-be54-ed5cd90134fa admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:40.634710 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c7853b3-f869-46e3-be54-ed5cd90134fa admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:40.639404 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a379ccee-eec8-4b3b-acdd-4b784d3e4c00 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:40.640062 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a379ccee-eec8-4b3b-acdd-4b784d3e4c00 admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:40.640343 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a379ccee-eec8-4b3b-acdd-4b784d3e4c00 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:40.640619 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a379ccee-eec8-4b3b-acdd-4b784d3e4c00 admin admin] PATH_INFO: `/v3/projects/a68704962f7346998beb980b920bbbb2/users/5121e8ae28b743dbaef18f5ed9a82b33/roles/c51fe7c5ad804040879acfb54269455f` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:40.642598 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a379ccee-eec8-4b3b-acdd-4b784d3e4c00 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a68704962f7346998beb980b920bbbb2, user_id=5121e8ae28b743dbaef18f5ed9a82b33, role_id=c51fe7c5ad804040879acfb54269455f)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:40.645854 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a379ccee-eec8-4b3b-acdd-4b784d3e4c00 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:40.656016 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1240/2455] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:43:40 2022] PUT /identity/v3/projects/a68704962f7346998beb980b920bbbb2/users/5121e8ae28b743dbaef18f5ed9a82b33/roles/c51fe7c5ad804040879acfb54269455f => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:43:40.667022 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3374c8c-7d14-46d7-873b-0f6bfea91dee None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:40.667308 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3374c8c-7d14-46d7-873b-0f6bfea91dee None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:40.667608 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3374c8c-7d14-46d7-873b-0f6bfea91dee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:40.692478 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:43:40.695795 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-c3374c8c-7d14-46d7-873b-0f6bfea91dee None None] MFA Rules not processed for user `5121e8ae28b743dbaef18f5ed9a82b33`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:43:40.697699 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-c3374c8c-7d14-46d7-873b-0f6bfea91dee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:43:40.767918 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1216/2456] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:43:40 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:43:40.779503 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a379ccee-eec8-4b3b-acdd-4b784d3e4c00 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:40.781367 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-a379ccee-eec8-4b3b-acdd-4b784d3e4c00 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:43:40.792173 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:43:40.842100 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a379ccee-eec8-4b3b-acdd-4b784d3e4c00 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:40.845931 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2afa0ef-8cf4-4dff-8051-5e4bab78db00 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ffbc136c77b84e8c87128f7247f4ec03', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:40.846544 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2afa0ef-8cf4-4dff-8051-5e4bab78db00 service cinder] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:40.846826 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2afa0ef-8cf4-4dff-8051-5e4bab78db00 service cinder] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:40.847101 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2afa0ef-8cf4-4dff-8051-5e4bab78db00 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:40.903953 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2afa0ef-8cf4-4dff-8051-5e4bab78db00 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:40.905449 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2afa0ef-8cf4-4dff-8051-5e4bab78db00 service cinder] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:40.914913 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1241/2457] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:43:40 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 136 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:43:45.133568 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3374c8c-7d14-46d7-873b-0f6bfea91dee None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:45.134966 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-c3374c8c-7d14-46d7-873b-0f6bfea91dee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:43:45.175242 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3374c8c-7d14-46d7-873b-0f6bfea91dee None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:45.179589 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b008d08-0b6a-48df-85b5-650f33ae76bb service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ffbc136c77b84e8c87128f7247f4ec03', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:45.180189 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b008d08-0b6a-48df-85b5-650f33ae76bb service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:45.180468 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b008d08-0b6a-48df-85b5-650f33ae76bb service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:45.180771 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b008d08-0b6a-48df-85b5-650f33ae76bb service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:45.211828 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b008d08-0b6a-48df-85b5-650f33ae76bb service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:45.213920 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b008d08-0b6a-48df-85b5-650f33ae76bb service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:45.225562 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1217/2458] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:43:45 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:43:50.121353 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2afa0ef-8cf4-4dff-8051-5e4bab78db00 service cinder] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:50.219849 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2afa0ef-8cf4-4dff-8051-5e4bab78db00 service cinder] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:50.224601 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdfdb37d-0d33-44cc-aa99-ddc6c3a8f880 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:50.225327 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdfdb37d-0d33-44cc-aa99-ddc6c3a8f880 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:50.225609 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdfdb37d-0d33-44cc-aa99-ddc6c3a8f880 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:50.225881 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdfdb37d-0d33-44cc-aa99-ddc6c3a8f880 admin admin] PATH_INFO: `/v3/users/33ee94fa37f04587a15460d867d9cb56` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:50.227015 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdfdb37d-0d33-44cc-aa99-ddc6c3a8f880 admin admin] RBAC: Authorizing `identity:delete_user(user_id=33ee94fa37f04587a15460d867d9cb56)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:50.228981 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdfdb37d-0d33-44cc-aa99-ddc6c3a8f880 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:50.257290 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:43:50.275663 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-fdfdb37d-0d33-44cc-aa99-ddc6c3a8f880 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '33ee94fa37f04587a15460d867d9cb56'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:50.282670 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-fdfdb37d-0d33-44cc-aa99-ddc6c3a8f880 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '33ee94fa37f04587a15460d867d9cb56'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:50.294692 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-fdfdb37d-0d33-44cc-aa99-ddc6c3a8f880 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '33ee94fa37f04587a15460d867d9cb56'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:50.295400 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-fdfdb37d-0d33-44cc-aa99-ddc6c3a8f880 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '33ee94fa37f04587a15460d867d9cb56'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:50.303135 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-fdfdb37d-0d33-44cc-aa99-ddc6c3a8f880 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '33ee94fa37f04587a15460d867d9cb56'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:50.309117 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1242/2459] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:43:50 2022] DELETE /identity/v3/users/33ee94fa37f04587a15460d867d9cb56 => generated 0 bytes in 188 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:43:50.313584 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2abc0fb-6849-45a7-9f1c-40c227444368 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:50.314081 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2abc0fb-6849-45a7-9f1c-40c227444368 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:50.314188 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2abc0fb-6849-45a7-9f1c-40c227444368 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:50.359013 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-b2abc0fb-6849-45a7-9f1c-40c227444368 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:43:50.360154 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-b2abc0fb-6849-45a7-9f1c-40c227444368 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:43:50.418351 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1218/2460] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:43:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:43:50.428412 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdfdb37d-0d33-44cc-aa99-ddc6c3a8f880 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:50.429909 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-fdfdb37d-0d33-44cc-aa99-ddc6c3a8f880 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:43:50.494178 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdfdb37d-0d33-44cc-aa99-ddc6c3a8f880 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:50.498142 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b0ef5f2-d5b5-485b-ac80-9447ef06faa1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:50.498788 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b0ef5f2-d5b5-485b-ac80-9447ef06faa1 service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:50.499073 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b0ef5f2-d5b5-485b-ac80-9447ef06faa1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:50.499346 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b0ef5f2-d5b5-485b-ac80-9447ef06faa1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:50.509538 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b0ef5f2-d5b5-485b-ac80-9447ef06faa1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:50.511030 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b0ef5f2-d5b5-485b-ac80-9447ef06faa1 service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:50.520826 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1243/2461] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:43:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:43:50.780523 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2abc0fb-6849-45a7-9f1c-40c227444368 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:50.781971 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-b2abc0fb-6849-45a7-9f1c-40c227444368 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:43:50.799054 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2abc0fb-6849-45a7-9f1c-40c227444368 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:50.806386 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a70638c4-d821-4516-ae98-2a4028d056b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:50.806986 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a70638c4-d821-4516-ae98-2a4028d056b1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:50.807264 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a70638c4-d821-4516-ae98-2a4028d056b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:50.807534 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a70638c4-d821-4516-ae98-2a4028d056b1 admin admin] PATH_INFO: `/v3/projects/9cac1a31801a470f83d857e094caed5a` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:50.808849 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a70638c4-d821-4516-ae98-2a4028d056b1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9cac1a31801a470f83d857e094caed5a)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:50.810954 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a70638c4-d821-4516-ae98-2a4028d056b1 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:50.827059 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:43:50.858942 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-a70638c4-d821-4516-ae98-2a4028d056b1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9cac1a31801a470f83d857e094caed5a', 'request_id': 'req-a70638c4-d821-4516-ae98-2a4028d056b1'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:50.866407 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-a70638c4-d821-4516-ae98-2a4028d056b1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9cac1a31801a470f83d857e094caed5a', 'request_id': 'req-a70638c4-d821-4516-ae98-2a4028d056b1'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:50.872101 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-a70638c4-d821-4516-ae98-2a4028d056b1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9cac1a31801a470f83d857e094caed5a', 'request_id': 'req-a70638c4-d821-4516-ae98-2a4028d056b1'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:50.883141 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-a70638c4-d821-4516-ae98-2a4028d056b1 admin admin] The token cache is being invalidate because project 9cac1a31801a470f83d857e094caed5a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:43:50.883432 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-a70638c4-d821-4516-ae98-2a4028d056b1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:50.884064 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-a70638c4-d821-4516-ae98-2a4028d056b1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:50.885594 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1219/2462] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:43:50 2022] DELETE /identity/v3/projects/9cac1a31801a470f83d857e094caed5a => generated 0 bytes in 106 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:43:50.889727 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b0ef5f2-d5b5-485b-ac80-9447ef06faa1 service neutron] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:50.899609 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b0ef5f2-d5b5-485b-ac80-9447ef06faa1 service neutron] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:50.955455 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7582323-0a3d-427f-a468-b9c49e220ae0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:50.956080 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7582323-0a3d-427f-a468-b9c49e220ae0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:50.956358 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7582323-0a3d-427f-a468-b9c49e220ae0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:50.956629 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7582323-0a3d-427f-a468-b9c49e220ae0 admin admin] PATH_INFO: `/v3/users/5121e8ae28b743dbaef18f5ed9a82b33` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:50.957820 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7582323-0a3d-427f-a468-b9c49e220ae0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5121e8ae28b743dbaef18f5ed9a82b33)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:50.959686 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7582323-0a3d-427f-a468-b9c49e220ae0 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:51.004266 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-c7582323-0a3d-427f-a468-b9c49e220ae0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5121e8ae28b743dbaef18f5ed9a82b33'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:51.006749 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:43:51.012203 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-c7582323-0a3d-427f-a468-b9c49e220ae0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5121e8ae28b743dbaef18f5ed9a82b33'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:51.026226 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-c7582323-0a3d-427f-a468-b9c49e220ae0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5121e8ae28b743dbaef18f5ed9a82b33'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:51.026864 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-c7582323-0a3d-427f-a468-b9c49e220ae0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5121e8ae28b743dbaef18f5ed9a82b33'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:51.033607 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-c7582323-0a3d-427f-a468-b9c49e220ae0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5121e8ae28b743dbaef18f5ed9a82b33'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:51.039956 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1244/2463] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:43:50 2022] DELETE /identity/v3/users/5121e8ae28b743dbaef18f5ed9a82b33 => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 04:43:51.315397 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a70638c4-d821-4516-ae98-2a4028d056b1 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:51.316984 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-a70638c4-d821-4516-ae98-2a4028d056b1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:43:51.334341 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a70638c4-d821-4516-ae98-2a4028d056b1 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:51.392381 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c54e6125-6c10-47e7-8b12-c0ac7b45b936 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:51.392969 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c54e6125-6c10-47e7-8b12-c0ac7b45b936 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:51.393247 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c54e6125-6c10-47e7-8b12-c0ac7b45b936 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:51.393518 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c54e6125-6c10-47e7-8b12-c0ac7b45b936 admin admin] PATH_INFO: `/v3/projects/a68704962f7346998beb980b920bbbb2` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:51.395313 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c54e6125-6c10-47e7-8b12-c0ac7b45b936 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a68704962f7346998beb980b920bbbb2)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:51.397434 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c54e6125-6c10-47e7-8b12-c0ac7b45b936 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:51.443017 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-c54e6125-6c10-47e7-8b12-c0ac7b45b936 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a68704962f7346998beb980b920bbbb2', 'request_id': 'req-c54e6125-6c10-47e7-8b12-c0ac7b45b936'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:51.450316 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-c54e6125-6c10-47e7-8b12-c0ac7b45b936 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a68704962f7346998beb980b920bbbb2', 'request_id': 'req-c54e6125-6c10-47e7-8b12-c0ac7b45b936'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:51.456009 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-c54e6125-6c10-47e7-8b12-c0ac7b45b936 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a68704962f7346998beb980b920bbbb2', 'request_id': 'req-c54e6125-6c10-47e7-8b12-c0ac7b45b936'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:51.468148 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-c54e6125-6c10-47e7-8b12-c0ac7b45b936 admin admin] The token cache is being invalidate because project a68704962f7346998beb980b920bbbb2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:43:51.468444 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-c54e6125-6c10-47e7-8b12-c0ac7b45b936 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:51.469200 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-c54e6125-6c10-47e7-8b12-c0ac7b45b936 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:43:51.470705 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1220/2464] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:43:51 2022] DELETE /identity/v3/projects/a68704962f7346998beb980b920bbbb2 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:43:51.476925 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f84598d3-1672-46bb-88d2-4e406660695e None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:51.477433 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f84598d3-1672-46bb-88d2-4e406660695e None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:51.477549 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f84598d3-1672-46bb-88d2-4e406660695e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:51.525970 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-f84598d3-1672-46bb-88d2-4e406660695e None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:43:51.527170 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-f84598d3-1672-46bb-88d2-4e406660695e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:43:51.584826 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1245/2465] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:43:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:43:51.647330 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91df047c-9305-4b1a-b444-16e6c9b0eecf None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:51.647542 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91df047c-9305-4b1a-b444-16e6c9b0eecf None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:51.647784 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91df047c-9305-4b1a-b444-16e6c9b0eecf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:51.668466 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-91df047c-9305-4b1a-b444-16e6c9b0eecf None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:43:51.669690 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-91df047c-9305-4b1a-b444-16e6c9b0eecf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:43:51.677877 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1221/2466] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:43:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:43:51.682352 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f84598d3-1672-46bb-88d2-4e406660695e None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:51.693203 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f84598d3-1672-46bb-88d2-4e406660695e None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:51.698433 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f8d05a4-b127-454a-b2ac-dd3bf4e75ad4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:51.699032 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f8d05a4-b127-454a-b2ac-dd3bf4e75ad4 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:51.699309 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f8d05a4-b127-454a-b2ac-dd3bf4e75ad4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:51.699578 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f8d05a4-b127-454a-b2ac-dd3bf4e75ad4 admin admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:51.700243 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f8d05a4-b127-454a-b2ac-dd3bf4e75ad4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:43:51.700548 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f8d05a4-b127-454a-b2ac-dd3bf4e75ad4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:51.702200 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f8d05a4-b127-454a-b2ac-dd3bf4e75ad4 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:51.714939 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1246/2467] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Sep 25 04:43:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:43:51.719417 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91df047c-9305-4b1a-b444-16e6c9b0eecf None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:51.728794 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91df047c-9305-4b1a-b444-16e6c9b0eecf None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:51.733414 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6be5174b-a6de-465c-b635-a2244a97732a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:51.733999 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6be5174b-a6de-465c-b635-a2244a97732a admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:51.734275 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6be5174b-a6de-465c-b635-a2244a97732a admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:51.734544 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6be5174b-a6de-465c-b635-a2244a97732a admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:51.735173 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6be5174b-a6de-465c-b635-a2244a97732a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:51.737081 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6be5174b-a6de-465c-b635-a2244a97732a admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:51.763482 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1222/2468] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:43:51 2022] POST /identity/v3/projects => generated 373 bytes in 44 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:43:51.767690 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f8d05a4-b127-454a-b2ac-dd3bf4e75ad4 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:51.779195 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f8d05a4-b127-454a-b2ac-dd3bf4e75ad4 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:51.835926 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3c8cc37-09bf-4113-99d6-63e4abf461e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:51.836546 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3c8cc37-09bf-4113-99d6-63e4abf461e8 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:51.836864 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3c8cc37-09bf-4113-99d6-63e4abf461e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:51.837142 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3c8cc37-09bf-4113-99d6-63e4abf461e8 admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:51.837785 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3c8cc37-09bf-4113-99d6-63e4abf461e8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:51.839973 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3c8cc37-09bf-4113-99d6-63e4abf461e8 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:51.864358 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1247/2469] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:43:51 2022] POST /identity/v3/users => generated 287 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 04:43:51.867614 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6be5174b-a6de-465c-b635-a2244a97732a admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:51.874362 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=83,keystone:DELETE=14,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:43:51.878375 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6be5174b-a6de-465c-b635-a2244a97732a admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:51.886433 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e8c0c30-328d-45c3-b3be-ae4995176257 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:51.887695 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e8c0c30-328d-45c3-b3be-ae4995176257 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:51.888203 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e8c0c30-328d-45c3-b3be-ae4995176257 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:51.888596 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e8c0c30-328d-45c3-b3be-ae4995176257 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:51.889487 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e8c0c30-328d-45c3-b3be-ae4995176257 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:51.891110 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e8c0c30-328d-45c3-b3be-ae4995176257 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:51.902320 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1223/2470] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:43:51 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:43:51.906191 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3c8cc37-09bf-4113-99d6-63e4abf461e8 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:51.916343 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3c8cc37-09bf-4113-99d6-63e4abf461e8 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:51.921129 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5a89a11-5fca-45c0-be36-68725c8337e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:51.921732 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a89a11-5fca-45c0-be36-68725c8337e4 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:51.922015 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a89a11-5fca-45c0-be36-68725c8337e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:51.922289 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a89a11-5fca-45c0-be36-68725c8337e4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:51.922950 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5a89a11-5fca-45c0-be36-68725c8337e4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:51.924544 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5a89a11-5fca-45c0-be36-68725c8337e4 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:51.935744 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1248/2471] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:43:51 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:43:51.939891 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e8c0c30-328d-45c3-b3be-ae4995176257 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:51.950974 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e8c0c30-328d-45c3-b3be-ae4995176257 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:51.955688 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9912c232-ff90-438d-a58b-6147d0f84980 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:51.956354 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9912c232-ff90-438d-a58b-6147d0f84980 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:51.956754 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9912c232-ff90-438d-a58b-6147d0f84980 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:51.956973 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9912c232-ff90-438d-a58b-6147d0f84980 admin admin] PATH_INFO: `/v3/projects/8fb362553bd743bfbbe783296f2597e0/users/cad7c7ccaeef409ebb5af348300068f5/roles/8f6cc574056c4278bc9778f14dc4b411` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:51.972570 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9912c232-ff90-438d-a58b-6147d0f84980 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8fb362553bd743bfbbe783296f2597e0, user_id=cad7c7ccaeef409ebb5af348300068f5, role_id=8f6cc574056c4278bc9778f14dc4b411)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:51.975448 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9912c232-ff90-438d-a58b-6147d0f84980 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:51.989830 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1224/2472] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:43:51 2022] PUT /identity/v3/projects/8fb362553bd743bfbbe783296f2597e0/users/cad7c7ccaeef409ebb5af348300068f5/roles/8f6cc574056c4278bc9778f14dc4b411 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:43:51.997124 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1517c864-42f5-4314-9dc9-c1baecefa904 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:51.997416 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1517c864-42f5-4314-9dc9-c1baecefa904 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:51.997724 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1517c864-42f5-4314-9dc9-c1baecefa904 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:52.017326 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-1517c864-42f5-4314-9dc9-c1baecefa904 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:43:52.018727 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-1517c864-42f5-4314-9dc9-c1baecefa904 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:43:52.022999 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:DELETE=3,keystone:SELECT=94,keystone:INSERT=3 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:43:52.085075 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1249/2473] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:43:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:43:52.095186 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9912c232-ff90-438d-a58b-6147d0f84980 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:52.097061 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-9912c232-ff90-438d-a58b-6147d0f84980 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:43:52.155895 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9912c232-ff90-438d-a58b-6147d0f84980 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:52.159659 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad24effb-a090-4e1d-8347-ccc4643ae929 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:52.160862 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad24effb-a090-4e1d-8347-ccc4643ae929 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:52.161149 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad24effb-a090-4e1d-8347-ccc4643ae929 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:52.161423 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad24effb-a090-4e1d-8347-ccc4643ae929 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:52.170892 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad24effb-a090-4e1d-8347-ccc4643ae929 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:52.172318 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad24effb-a090-4e1d-8347-ccc4643ae929 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:52.182607 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1225/2474] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:43:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 25 04:43:52.829285 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3256f7cb-8f65-46b8-aa29-6184db127585 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:52.829801 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3256f7cb-8f65-46b8-aa29-6184db127585 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:52.829972 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3256f7cb-8f65-46b8-aa29-6184db127585 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:52.861299 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-3256f7cb-8f65-46b8-aa29-6184db127585 None None] MFA Rules not processed for user `cad7c7ccaeef409ebb5af348300068f5`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:43:52.862659 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-3256f7cb-8f65-46b8-aa29-6184db127585 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:43:52.930274 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1250/2475] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:43:52 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:43:52.941843 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad24effb-a090-4e1d-8347-ccc4643ae929 service neutron] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:52.943388 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-ad24effb-a090-4e1d-8347-ccc4643ae929 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:43:52.999989 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad24effb-a090-4e1d-8347-ccc4643ae929 service neutron] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:53.003525 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cf1e7c6-8510-4e96-b28c-e84e9e4799a2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ffbc136c77b84e8c87128f7247f4ec03', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:53.004778 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cf1e7c6-8510-4e96-b28c-e84e9e4799a2 service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:53.005061 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cf1e7c6-8510-4e96-b28c-e84e9e4799a2 service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:53.005330 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cf1e7c6-8510-4e96-b28c-e84e9e4799a2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:53.014219 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cf1e7c6-8510-4e96-b28c-e84e9e4799a2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:53.015650 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cf1e7c6-8510-4e96-b28c-e84e9e4799a2 service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:53.025397 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1226/2476] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:43:52 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:43:54.393914 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c590fe3-ecc8-48ad-9f1a-be8fc4afa3ce None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:54.395060 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c590fe3-ecc8-48ad-9f1a-be8fc4afa3ce None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:54.395244 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c590fe3-ecc8-48ad-9f1a-be8fc4afa3ce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:54.418960 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-7c590fe3-ecc8-48ad-9f1a-be8fc4afa3ce None None] MFA Rules not processed for user `cad7c7ccaeef409ebb5af348300068f5`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:43:54.420169 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-7c590fe3-ecc8-48ad-9f1a-be8fc4afa3ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:43:54.427432 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1251/2477] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:43:54 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:43:55.697478 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cf1e7c6-8510-4e96-b28c-e84e9e4799a2 service cinder] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:43:55.698956 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-8cf1e7c6-8510-4e96-b28c-e84e9e4799a2 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:43:55.782665 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:43:55.793763 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cf1e7c6-8510-4e96-b28c-e84e9e4799a2 service cinder] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:43:55.799069 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f80eb05e-cc5c-4c3e-b6fd-13cc366b8d89 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '676d04ef5a984edcaad63d9156568660', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:43:55.799682 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f80eb05e-cc5c-4c3e-b6fd-13cc366b8d89 service placement] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:43:55.799964 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f80eb05e-cc5c-4c3e-b6fd-13cc366b8d89 service placement] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:43:55.800238 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f80eb05e-cc5c-4c3e-b6fd-13cc366b8d89 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:43:55.802140 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-f80eb05e-cc5c-4c3e-b6fd-13cc366b8d89 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:43:55.814842 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f80eb05e-cc5c-4c3e-b6fd-13cc366b8d89 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:43:55.817118 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f80eb05e-cc5c-4c3e-b6fd-13cc366b8d89 service placement] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:43:55.827747 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1227/2478] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:43:55 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 131 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:44:04.408536 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=58 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:44:05.823568 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=7 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:44:09.295712 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c590fe3-ecc8-48ad-9f1a-be8fc4afa3ce None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:44:09.297132 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-7c590fe3-ecc8-48ad-9f1a-be8fc4afa3ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:44:09.329717 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c590fe3-ecc8-48ad-9f1a-be8fc4afa3ce None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:44:09.333805 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8b0469e-0247-4ef6-b6f7-b87f3c2b2073 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ffbc136c77b84e8c87128f7247f4ec03', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:44:09.334399 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8b0469e-0247-4ef6-b6f7-b87f3c2b2073 service cinder] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:44:09.334674 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8b0469e-0247-4ef6-b6f7-b87f3c2b2073 service cinder] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:44:09.334938 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8b0469e-0247-4ef6-b6f7-b87f3c2b2073 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:44:09.336778 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-f8b0469e-0247-4ef6-b6f7-b87f3c2b2073 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:44:09.396207 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8b0469e-0247-4ef6-b6f7-b87f3c2b2073 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:44:09.397676 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8b0469e-0247-4ef6-b6f7-b87f3c2b2073 service cinder] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:44:09.406728 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1252/2479] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:44:09 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:44:16.211614 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c3357c4-3143-45bc-967f-d4c80760311e None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:44:16.212096 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c3357c4-3143-45bc-967f-d4c80760311e None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:44:16.212199 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c3357c4-3143-45bc-967f-d4c80760311e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:44:16.231694 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-5c3357c4-3143-45bc-967f-d4c80760311e None None] MFA Rules not processed for user `cad7c7ccaeef409ebb5af348300068f5`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:44:16.232929 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-5c3357c4-3143-45bc-967f-d4c80760311e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:44:16.239995 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1228/2480] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:44:16 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:44:19.402497 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=37 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:44:20.485002 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb454e61-39fe-477e-8f64-81d00ea2aba9 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:44:20.485573 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb454e61-39fe-477e-8f64-81d00ea2aba9 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:44:20.485573 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb454e61-39fe-477e-8f64-81d00ea2aba9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:44:20.505906 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-cb454e61-39fe-477e-8f64-81d00ea2aba9 None None] MFA Rules not processed for user `329e3f9810b648218dd45a8b489adb1d`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:44:20.507358 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-cb454e61-39fe-477e-8f64-81d00ea2aba9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:44:20.514469 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1253/2481] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:44:20 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:44:26.226902 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:44:30.499456 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=3 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:44:38.106636 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c3357c4-3143-45bc-967f-d4c80760311e None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:44:38.125869 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c3357c4-3143-45bc-967f-d4c80760311e None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:44:38.129560 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c6f4523-bba0-4229-8037-0b257271ae0a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:44:38.130165 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c6f4523-bba0-4229-8037-0b257271ae0a service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:44:38.130446 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c6f4523-bba0-4229-8037-0b257271ae0a service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:44:38.130719 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c6f4523-bba0-4229-8037-0b257271ae0a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:44:38.133148 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-7c6f4523-bba0-4229-8037-0b257271ae0a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:44:38.198775 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c6f4523-bba0-4229-8037-0b257271ae0a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:44:38.200570 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c6f4523-bba0-4229-8037-0b257271ae0a service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:44:38.215155 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1229/2482] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:44:38 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:44:46.685373 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-615ca8c2-9024-4d6f-b042-8cefe09c5b7a None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:44:46.685862 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-615ca8c2-9024-4d6f-b042-8cefe09c5b7a None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:44:46.686066 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-615ca8c2-9024-4d6f-b042-8cefe09c5b7a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:44:46.717648 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-615ca8c2-9024-4d6f-b042-8cefe09c5b7a None None] MFA Rules not processed for user `050b3aab44bb4b17aa4a1ba7cccdb231`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:44:46.718865 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-615ca8c2-9024-4d6f-b042-8cefe09c5b7a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:44:46.785025 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1254/2483] 10.222.0.25 () {58 vars in 1009 bytes} [Sun Sep 25 04:44:46 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:44:46.838221 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c6f4523-bba0-4229-8037-0b257271ae0a service neutron] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:44:46.839929 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-7c6f4523-bba0-4229-8037-0b257271ae0a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:44:46.921914 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c6f4523-bba0-4229-8037-0b257271ae0a service neutron] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:44:46.927272 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e76799c-dd7c-43a5-b3ce-9f5b8e821742 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '23ea88f46e7740d295148a93ade29dd9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:44:46.927695 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e76799c-dd7c-43a5-b3ce-9f5b8e821742 service swift] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:44:46.928009 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e76799c-dd7c-43a5-b3ce-9f5b8e821742 service swift] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:44:46.928966 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e76799c-dd7c-43a5-b3ce-9f5b8e821742 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:44:46.937191 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e76799c-dd7c-43a5-b3ce-9f5b8e821742 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:44:46.938778 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e76799c-dd7c-43a5-b3ce-9f5b8e821742 service swift] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:44:46.946870 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1230/2484] 10.222.0.25 () {60 vars in 1384 bytes} [Sun Sep 25 04:44:46 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 25 04:44:56.558487 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9486ec4d-5e78-454e-8b49-c309e6d002c7 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:44:56.559075 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9486ec4d-5e78-454e-8b49-c309e6d002c7 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:44:56.559075 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9486ec4d-5e78-454e-8b49-c309e6d002c7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:44:56.580405 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-9486ec4d-5e78-454e-8b49-c309e6d002c7 None None] MFA Rules not processed for user `c50ba6f9a164463eae9a248936e7bf80`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:44:56.582104 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-9486ec4d-5e78-454e-8b49-c309e6d002c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:44:56.648295 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1255/2485] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:44:56 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:44:56.944434 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=66 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:45:03.648608 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-046aa442-b3dc-404c-ba5b-e46f510e1a4e None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:45:03.649210 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-046aa442-b3dc-404c-ba5b-e46f510e1a4e None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:45:03.649210 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-046aa442-b3dc-404c-ba5b-e46f510e1a4e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:45:03.668729 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-046aa442-b3dc-404c-ba5b-e46f510e1a4e None None] MFA Rules not processed for user `050b3aab44bb4b17aa4a1ba7cccdb231`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:45:03.669935 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-046aa442-b3dc-404c-ba5b-e46f510e1a4e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:45:03.677180 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1231/2486] 10.222.0.25 () {58 vars in 967 bytes} [Sun Sep 25 04:45:03 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:45:03.690069 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9486ec4d-5e78-454e-8b49-c309e6d002c7 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:45:03.702061 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9486ec4d-5e78-454e-8b49-c309e6d002c7 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:45:03.707054 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3f7c9a3-c100-44b9-b92b-916a91618194 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '23ea88f46e7740d295148a93ade29dd9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:45:03.707654 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3f7c9a3-c100-44b9-b92b-916a91618194 service swift] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:45:03.707934 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3f7c9a3-c100-44b9-b92b-916a91618194 service swift] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:45:03.708205 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3f7c9a3-c100-44b9-b92b-916a91618194 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:45:03.717095 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3f7c9a3-c100-44b9-b92b-916a91618194 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:45:03.718656 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3f7c9a3-c100-44b9-b92b-916a91618194 service swift] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:45:03.727377 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1256/2487] 10.222.0.25 () {60 vars in 1384 bytes} [Sun Sep 25 04:45:03 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 25 04:45:05.625454 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-046aa442-b3dc-404c-ba5b-e46f510e1a4e None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:45:05.637491 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-046aa442-b3dc-404c-ba5b-e46f510e1a4e None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:45:05.642261 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95f9ea75-6b7b-476e-9430-3745b9047ec1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:45:05.642901 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95f9ea75-6b7b-476e-9430-3745b9047ec1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:45:05.643181 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95f9ea75-6b7b-476e-9430-3745b9047ec1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:45:05.643456 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95f9ea75-6b7b-476e-9430-3745b9047ec1 admin admin] PATH_INFO: `/v3/users/cad7c7ccaeef409ebb5af348300068f5` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:45:05.644569 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95f9ea75-6b7b-476e-9430-3745b9047ec1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=cad7c7ccaeef409ebb5af348300068f5)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:45:05.646528 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95f9ea75-6b7b-476e-9430-3745b9047ec1 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:45:05.695216 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-95f9ea75-6b7b-476e-9430-3745b9047ec1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cad7c7ccaeef409ebb5af348300068f5'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:45:05.706023 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-95f9ea75-6b7b-476e-9430-3745b9047ec1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cad7c7ccaeef409ebb5af348300068f5'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:45:05.719488 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-95f9ea75-6b7b-476e-9430-3745b9047ec1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cad7c7ccaeef409ebb5af348300068f5'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:45:05.720277 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-95f9ea75-6b7b-476e-9430-3745b9047ec1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cad7c7ccaeef409ebb5af348300068f5'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:45:05.728027 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-95f9ea75-6b7b-476e-9430-3745b9047ec1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cad7c7ccaeef409ebb5af348300068f5'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:45:05.734110 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1232/2488] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:45:05 2022] DELETE /identity/v3/users/cad7c7ccaeef409ebb5af348300068f5 => generated 0 bytes in 109 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:45:05.916055 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3f7c9a3-c100-44b9-b92b-916a91618194 service swift] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:45:05.929631 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3f7c9a3-c100-44b9-b92b-916a91618194 service swift] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:45:05.988516 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dc682c8-36ef-4984-ab28-35561b103075 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:45:05.989179 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc682c8-36ef-4984-ab28-35561b103075 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:45:05.989462 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc682c8-36ef-4984-ab28-35561b103075 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:45:05.989736 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc682c8-36ef-4984-ab28-35561b103075 admin admin] PATH_INFO: `/v3/projects/8fb362553bd743bfbbe783296f2597e0` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:45:05.991545 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dc682c8-36ef-4984-ab28-35561b103075 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8fb362553bd743bfbbe783296f2597e0)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:45:05.993479 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dc682c8-36ef-4984-ab28-35561b103075 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:45:06.017890 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=99,keystone:DELETE=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:45:06.040839 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-2dc682c8-36ef-4984-ab28-35561b103075 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8fb362553bd743bfbbe783296f2597e0', 'request_id': 'req-2dc682c8-36ef-4984-ab28-35561b103075'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:45:06.048036 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-2dc682c8-36ef-4984-ab28-35561b103075 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8fb362553bd743bfbbe783296f2597e0', 'request_id': 'req-2dc682c8-36ef-4984-ab28-35561b103075'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:45:06.054140 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-2dc682c8-36ef-4984-ab28-35561b103075 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8fb362553bd743bfbbe783296f2597e0', 'request_id': 'req-2dc682c8-36ef-4984-ab28-35561b103075'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:45:06.064752 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-2dc682c8-36ef-4984-ab28-35561b103075 admin admin] The token cache is being invalidate because project 8fb362553bd743bfbbe783296f2597e0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81434) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:45:06.065040 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-2dc682c8-36ef-4984-ab28-35561b103075 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:45:06.065664 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-2dc682c8-36ef-4984-ab28-35561b103075 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:45:06.067168 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1257/2489] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:45:05 2022] DELETE /identity/v3/projects/8fb362553bd743bfbbe783296f2597e0 => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:45:06.072298 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-016e17cf-176d-44a3-b7ff-bab8a87c8eff None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:45:06.072596 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-016e17cf-176d-44a3-b7ff-bab8a87c8eff None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:45:06.072916 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-016e17cf-176d-44a3-b7ff-bab8a87c8eff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:45:06.093281 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-016e17cf-176d-44a3-b7ff-bab8a87c8eff None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:45:06.094495 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-016e17cf-176d-44a3-b7ff-bab8a87c8eff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:45:06.154354 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1233/2490] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:45:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:45:06.163459 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf1ea8d6-5f2d-4177-b319-c5e9a85c056b None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:45:06.163760 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf1ea8d6-5f2d-4177-b319-c5e9a85c056b None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:45:06.164034 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf1ea8d6-5f2d-4177-b319-c5e9a85c056b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:45:06.183855 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-bf1ea8d6-5f2d-4177-b319-c5e9a85c056b None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:45:06.185120 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-bf1ea8d6-5f2d-4177-b319-c5e9a85c056b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:45:06.193278 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1258/2491] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:45:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:45:06.197610 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-016e17cf-176d-44a3-b7ff-bab8a87c8eff None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:45:06.207263 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-016e17cf-176d-44a3-b7ff-bab8a87c8eff None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:45:06.212584 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd0fb21f-b710-437c-9daa-ff596fdc998d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:45:06.213200 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd0fb21f-b710-437c-9daa-ff596fdc998d admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:45:06.213480 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd0fb21f-b710-437c-9daa-ff596fdc998d admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:45:06.213754 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd0fb21f-b710-437c-9daa-ff596fdc998d admin admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:45:06.214652 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd0fb21f-b710-437c-9daa-ff596fdc998d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:45:06.214955 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd0fb21f-b710-437c-9daa-ff596fdc998d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:45:06.216548 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd0fb21f-b710-437c-9daa-ff596fdc998d admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:45:06.229361 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1234/2492] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Sep 25 04:45:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:45:06.233697 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf1ea8d6-5f2d-4177-b319-c5e9a85c056b None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:45:06.243411 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf1ea8d6-5f2d-4177-b319-c5e9a85c056b None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:45:06.248433 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f50b48a8-0d6e-461f-8649-aa5312360668 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:45:06.249065 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f50b48a8-0d6e-461f-8649-aa5312360668 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:45:06.249343 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f50b48a8-0d6e-461f-8649-aa5312360668 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:45:06.249610 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f50b48a8-0d6e-461f-8649-aa5312360668 admin admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:45:06.250230 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f50b48a8-0d6e-461f-8649-aa5312360668 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:45:06.252130 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f50b48a8-0d6e-461f-8649-aa5312360668 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:45:06.279918 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1259/2493] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:45:06 2022] POST /identity/v3/projects => generated 369 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 04:45:06.284000 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd0fb21f-b710-437c-9daa-ff596fdc998d admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:45:06.294662 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd0fb21f-b710-437c-9daa-ff596fdc998d admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:45:06.344253 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:45:06.355175 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bc72549-a3ec-43bc-b264-6e98f09db611 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:45:06.355995 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bc72549-a3ec-43bc-b264-6e98f09db611 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:45:06.356277 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bc72549-a3ec-43bc-b264-6e98f09db611 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:45:06.356553 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bc72549-a3ec-43bc-b264-6e98f09db611 admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:45:06.357224 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bc72549-a3ec-43bc-b264-6e98f09db611 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:45:06.359149 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bc72549-a3ec-43bc-b264-6e98f09db611 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:45:06.379781 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1235/2494] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:45:06 2022] POST /identity/v3/users => generated 285 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 04:45:06.384727 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f50b48a8-0d6e-461f-8649-aa5312360668 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:45:06.395013 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f50b48a8-0d6e-461f-8649-aa5312360668 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:45:06.400406 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8768661-6393-4be2-805d-54b93cd5bbfe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:45:06.401042 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8768661-6393-4be2-805d-54b93cd5bbfe admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:45:06.401320 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8768661-6393-4be2-805d-54b93cd5bbfe admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:45:06.401586 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8768661-6393-4be2-805d-54b93cd5bbfe admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:45:06.402218 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8768661-6393-4be2-805d-54b93cd5bbfe admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:45:06.403809 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8768661-6393-4be2-805d-54b93cd5bbfe admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:45:06.413515 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1260/2495] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:45:06 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:45:06.417475 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bc72549-a3ec-43bc-b264-6e98f09db611 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:45:06.427393 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bc72549-a3ec-43bc-b264-6e98f09db611 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:45:06.432365 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6c63cb5-29d0-42aa-97be-8e3e72d66cc5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:45:06.432992 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6c63cb5-29d0-42aa-97be-8e3e72d66cc5 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:45:06.433277 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6c63cb5-29d0-42aa-97be-8e3e72d66cc5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:45:06.433554 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6c63cb5-29d0-42aa-97be-8e3e72d66cc5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:45:06.434246 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6c63cb5-29d0-42aa-97be-8e3e72d66cc5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:45:06.435785 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6c63cb5-29d0-42aa-97be-8e3e72d66cc5 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:45:06.446933 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1236/2496] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:45:06 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:45:06.451317 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8768661-6393-4be2-805d-54b93cd5bbfe admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:45:06.461912 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8768661-6393-4be2-805d-54b93cd5bbfe admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:45:06.466841 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50cefb98-d5c1-410d-a3a9-b93bf16dffec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:45:06.467491 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50cefb98-d5c1-410d-a3a9-b93bf16dffec admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:45:06.467768 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50cefb98-d5c1-410d-a3a9-b93bf16dffec admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:45:06.468039 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50cefb98-d5c1-410d-a3a9-b93bf16dffec admin admin] PATH_INFO: `/v3/projects/b621634728944b2a9be669e1aaffa962/users/a71cba45bfab46d58c67eb1179fc574c/roles/8f6cc574056c4278bc9778f14dc4b411` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:45:06.483289 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50cefb98-d5c1-410d-a3a9-b93bf16dffec admin admin] RBAC: Authorizing `identity:create_grant(project_id=b621634728944b2a9be669e1aaffa962, user_id=a71cba45bfab46d58c67eb1179fc574c, role_id=8f6cc574056c4278bc9778f14dc4b411)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:45:06.486002 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50cefb98-d5c1-410d-a3a9-b93bf16dffec admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:45:06.499764 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1261/2497] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:45:06 2022] PUT /identity/v3/projects/b621634728944b2a9be669e1aaffa962/users/a71cba45bfab46d58c67eb1179fc574c/roles/8f6cc574056c4278bc9778f14dc4b411 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:45:06.510285 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-182d572c-211a-46b9-8e0b-ad41987ce7e4 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:45:06.510587 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-182d572c-211a-46b9-8e0b-ad41987ce7e4 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:45:06.510839 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-182d572c-211a-46b9-8e0b-ad41987ce7e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:45:06.538704 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-182d572c-211a-46b9-8e0b-ad41987ce7e4 None None] MFA Rules not processed for user `a71cba45bfab46d58c67eb1179fc574c`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:45:06.539923 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-182d572c-211a-46b9-8e0b-ad41987ce7e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:45:06.604092 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1237/2498] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:45:06 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:45:06.615801 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50cefb98-d5c1-410d-a3a9-b93bf16dffec admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:45:06.617700 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-50cefb98-d5c1-410d-a3a9-b93bf16dffec admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:45:06.675021 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50cefb98-d5c1-410d-a3a9-b93bf16dffec admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:45:06.679181 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b32fc6f-85c2-4ea9-a719-5bf3b5f24d79 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '316eaa67d5464091af47efd489b96b87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:45:06.679799 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b32fc6f-85c2-4ea9-a719-5bf3b5f24d79 service glance] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:45:06.680083 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b32fc6f-85c2-4ea9-a719-5bf3b5f24d79 service glance] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:45:06.680356 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b32fc6f-85c2-4ea9-a719-5bf3b5f24d79 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:45:06.689073 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b32fc6f-85c2-4ea9-a719-5bf3b5f24d79 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:45:06.690502 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b32fc6f-85c2-4ea9-a719-5bf3b5f24d79 service glance] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:45:06.700159 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1262/2499] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:45:06 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:45:16.601121 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=42,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:45:16.695949 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=68,keystone:DELETE=6,keystone:INSERT=3 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:45:44.747086 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dc6593b-a291-4c04-99af-342c388ea459 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:45:44.748930 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dc6593b-a291-4c04-99af-342c388ea459 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:45:44.749509 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dc6593b-a291-4c04-99af-342c388ea459 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:45:44.770081 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-4dc6593b-a291-4c04-99af-342c388ea459 None None] MFA Rules not processed for user `050b3aab44bb4b17aa4a1ba7cccdb231`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:45:44.771375 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-4dc6593b-a291-4c04-99af-342c388ea459 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:45:44.831854 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1238/2500] 10.222.0.25 () {58 vars in 1009 bytes} [Sun Sep 25 04:45:44 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:45:44.845947 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b32fc6f-85c2-4ea9-a719-5bf3b5f24d79 service glance] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:45:44.847597 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-1b32fc6f-85c2-4ea9-a719-5bf3b5f24d79 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:45:44.926619 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b32fc6f-85c2-4ea9-a719-5bf3b5f24d79 service glance] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:45:44.933849 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-550dcac2-996a-485b-b886-7ca82496c330 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '23ea88f46e7740d295148a93ade29dd9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:45:44.933849 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-550dcac2-996a-485b-b886-7ca82496c330 service swift] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:45:44.934159 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-550dcac2-996a-485b-b886-7ca82496c330 service swift] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:45:44.934159 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-550dcac2-996a-485b-b886-7ca82496c330 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:45:44.942685 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-550dcac2-996a-485b-b886-7ca82496c330 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:45:44.944258 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-550dcac2-996a-485b-b886-7ca82496c330 service swift] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:45:44.952611 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1263/2501] 10.222.0.25 () {60 vars in 1384 bytes} [Sun Sep 25 04:45:44 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 25 04:45:48.272717 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dc6593b-a291-4c04-99af-342c388ea459 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:45:48.369697 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dc6593b-a291-4c04-99af-342c388ea459 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:45:48.374960 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8662aada-1a6a-4862-a01c-10bc727ce362 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:45:48.375605 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8662aada-1a6a-4862-a01c-10bc727ce362 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:45:48.375887 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8662aada-1a6a-4862-a01c-10bc727ce362 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:45:48.376161 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8662aada-1a6a-4862-a01c-10bc727ce362 admin admin] PATH_INFO: `/v3/users/a71cba45bfab46d58c67eb1179fc574c` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:45:48.377349 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8662aada-1a6a-4862-a01c-10bc727ce362 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a71cba45bfab46d58c67eb1179fc574c)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:45:48.380147 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8662aada-1a6a-4862-a01c-10bc727ce362 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:45:48.443756 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-8662aada-1a6a-4862-a01c-10bc727ce362 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a71cba45bfab46d58c67eb1179fc574c'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:45:48.454595 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-8662aada-1a6a-4862-a01c-10bc727ce362 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a71cba45bfab46d58c67eb1179fc574c'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:45:48.477772 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-8662aada-1a6a-4862-a01c-10bc727ce362 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a71cba45bfab46d58c67eb1179fc574c'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:45:48.479743 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-8662aada-1a6a-4862-a01c-10bc727ce362 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a71cba45bfab46d58c67eb1179fc574c'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:45:48.492106 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-8662aada-1a6a-4862-a01c-10bc727ce362 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a71cba45bfab46d58c67eb1179fc574c'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:45:48.501635 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1239/2502] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:45:48 2022] DELETE /identity/v3/users/a71cba45bfab46d58c67eb1179fc574c => generated 0 bytes in 229 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:45:48.507363 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7e77b42-bb8d-42c9-af8f-e619446bf598 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:45:48.508383 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7e77b42-bb8d-42c9-af8f-e619446bf598 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:45:48.508861 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7e77b42-bb8d-42c9-af8f-e619446bf598 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:45:48.546236 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-b7e77b42-bb8d-42c9-af8f-e619446bf598 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:45:48.547595 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-b7e77b42-bb8d-42c9-af8f-e619446bf598 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:45:48.625021 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1264/2503] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:45:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:45:48.637247 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8662aada-1a6a-4862-a01c-10bc727ce362 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:45:48.638994 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-8662aada-1a6a-4862-a01c-10bc727ce362 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:45:48.677400 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:45:48.737050 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8662aada-1a6a-4862-a01c-10bc727ce362 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:45:48.741586 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c835eb73-5772-4236-b7e7-a67b48bd7261 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:45:48.742208 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c835eb73-5772-4236-b7e7-a67b48bd7261 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:45:48.742545 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c835eb73-5772-4236-b7e7-a67b48bd7261 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:45:48.742914 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c835eb73-5772-4236-b7e7-a67b48bd7261 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:45:48.756226 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c835eb73-5772-4236-b7e7-a67b48bd7261 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:45:48.757717 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c835eb73-5772-4236-b7e7-a67b48bd7261 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:45:48.771007 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1240/2504] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:45:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 135 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:45:49.081619 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7e77b42-bb8d-42c9-af8f-e619446bf598 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:45:49.086094 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-b7e77b42-bb8d-42c9-af8f-e619446bf598 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:45:49.119694 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7e77b42-bb8d-42c9-af8f-e619446bf598 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:45:49.128966 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ea06bd6-b00a-4e9d-92aa-59ed33872398 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:45:49.129612 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ea06bd6-b00a-4e9d-92aa-59ed33872398 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:45:49.129887 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ea06bd6-b00a-4e9d-92aa-59ed33872398 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:45:49.130159 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ea06bd6-b00a-4e9d-92aa-59ed33872398 admin admin] PATH_INFO: `/v3/projects/b621634728944b2a9be669e1aaffa962` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:45:49.131339 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ea06bd6-b00a-4e9d-92aa-59ed33872398 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b621634728944b2a9be669e1aaffa962)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:45:49.133458 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ea06bd6-b00a-4e9d-92aa-59ed33872398 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:45:49.195826 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-4ea06bd6-b00a-4e9d-92aa-59ed33872398 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b621634728944b2a9be669e1aaffa962', 'request_id': 'req-4ea06bd6-b00a-4e9d-92aa-59ed33872398'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:45:49.207110 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-4ea06bd6-b00a-4e9d-92aa-59ed33872398 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b621634728944b2a9be669e1aaffa962', 'request_id': 'req-4ea06bd6-b00a-4e9d-92aa-59ed33872398'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:45:49.214376 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-4ea06bd6-b00a-4e9d-92aa-59ed33872398 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b621634728944b2a9be669e1aaffa962', 'request_id': 'req-4ea06bd6-b00a-4e9d-92aa-59ed33872398'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:45:49.227514 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-4ea06bd6-b00a-4e9d-92aa-59ed33872398 admin admin] The token cache is being invalidate because project b621634728944b2a9be669e1aaffa962 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81434) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:45:49.227817 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-4ea06bd6-b00a-4e9d-92aa-59ed33872398 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:45:49.229061 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-4ea06bd6-b00a-4e9d-92aa-59ed33872398 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:45:49.232401 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1265/2505] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:45:49 2022] DELETE /identity/v3/projects/b621634728944b2a9be669e1aaffa962 => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:45:49.237417 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42482aea-0564-49a8-8864-d2902c823028 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:45:49.237711 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42482aea-0564-49a8-8864-d2902c823028 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:45:49.237998 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42482aea-0564-49a8-8864-d2902c823028 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:45:49.268306 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-42482aea-0564-49a8-8864-d2902c823028 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:45:49.270023 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-42482aea-0564-49a8-8864-d2902c823028 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:45:49.392519 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1241/2506] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:45:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:45:49.407939 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e282f477-db22-4da9-b704-c985eb922688 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:45:49.408148 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e282f477-db22-4da9-b704-c985eb922688 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:45:49.408442 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e282f477-db22-4da9-b704-c985eb922688 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:45:49.443495 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-e282f477-db22-4da9-b704-c985eb922688 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:45:49.445180 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-e282f477-db22-4da9-b704-c985eb922688 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:45:49.464401 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1266/2507] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:45:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 59 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:45:49.469891 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42482aea-0564-49a8-8864-d2902c823028 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:45:49.493999 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42482aea-0564-49a8-8864-d2902c823028 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:45:49.498995 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-598c030e-17ff-4197-892d-ec3af0dcc424 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:45:49.499610 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-598c030e-17ff-4197-892d-ec3af0dcc424 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:45:49.499896 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-598c030e-17ff-4197-892d-ec3af0dcc424 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:45:49.500169 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-598c030e-17ff-4197-892d-ec3af0dcc424 admin admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:45:49.500861 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-598c030e-17ff-4197-892d-ec3af0dcc424 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:45:49.501200 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-598c030e-17ff-4197-892d-ec3af0dcc424 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:45:49.502818 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-598c030e-17ff-4197-892d-ec3af0dcc424 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:45:49.520695 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1242/2508] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Sep 25 04:45:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:45:49.522340 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e282f477-db22-4da9-b704-c985eb922688 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:45:49.537072 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e282f477-db22-4da9-b704-c985eb922688 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:45:49.543743 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8089f2dd-5222-48db-869a-34b58c3f98ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:45:49.544371 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8089f2dd-5222-48db-869a-34b58c3f98ff admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:45:49.544679 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8089f2dd-5222-48db-869a-34b58c3f98ff admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:45:49.544969 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8089f2dd-5222-48db-869a-34b58c3f98ff admin admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:45:49.545619 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8089f2dd-5222-48db-869a-34b58c3f98ff admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:45:49.547538 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8089f2dd-5222-48db-869a-34b58c3f98ff admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:45:49.566359 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:45:49.632705 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1267/2509] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:45:49 2022] POST /identity/v3/projects => generated 365 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:45:49.636953 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-598c030e-17ff-4197-892d-ec3af0dcc424 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:45:49.655552 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-598c030e-17ff-4197-892d-ec3af0dcc424 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:45:49.732992 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fec2741-27b4-495d-8308-f90bf7a59fd9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:45:49.734118 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fec2741-27b4-495d-8308-f90bf7a59fd9 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:45:49.734974 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fec2741-27b4-495d-8308-f90bf7a59fd9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:45:49.735447 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fec2741-27b4-495d-8308-f90bf7a59fd9 admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:45:49.736110 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fec2741-27b4-495d-8308-f90bf7a59fd9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:45:49.738097 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fec2741-27b4-495d-8308-f90bf7a59fd9 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:45:49.768177 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1243/2510] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:45:49 2022] POST /identity/v3/users => generated 283 bytes in 132 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 04:45:49.772086 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8089f2dd-5222-48db-869a-34b58c3f98ff admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:45:49.785015 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8089f2dd-5222-48db-869a-34b58c3f98ff admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:45:49.791357 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7662121a-8eac-4acd-ae51-be177b3b2abf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:45:49.791984 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7662121a-8eac-4acd-ae51-be177b3b2abf admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:45:49.792371 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7662121a-8eac-4acd-ae51-be177b3b2abf admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:45:49.792699 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7662121a-8eac-4acd-ae51-be177b3b2abf admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:45:49.793354 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7662121a-8eac-4acd-ae51-be177b3b2abf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:45:49.794967 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7662121a-8eac-4acd-ae51-be177b3b2abf admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:45:49.808806 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1268/2511] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:45:49 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:45:49.812857 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fec2741-27b4-495d-8308-f90bf7a59fd9 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:45:49.827781 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fec2741-27b4-495d-8308-f90bf7a59fd9 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:45:49.833250 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54d81424-e948-4518-8a02-3cc5e50ace0c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:45:49.833942 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d81424-e948-4518-8a02-3cc5e50ace0c admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:45:49.834399 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d81424-e948-4518-8a02-3cc5e50ace0c admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:45:49.834749 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d81424-e948-4518-8a02-3cc5e50ace0c admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:45:49.835472 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54d81424-e948-4518-8a02-3cc5e50ace0c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:45:49.837189 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54d81424-e948-4518-8a02-3cc5e50ace0c admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:45:49.856391 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1244/2512] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:45:49 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:45:49.857720 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7662121a-8eac-4acd-ae51-be177b3b2abf admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:45:49.868987 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7662121a-8eac-4acd-ae51-be177b3b2abf admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:45:49.875749 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-167adbf9-96fb-489e-9fea-3424465a9e78 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:45:49.876421 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-167adbf9-96fb-489e-9fea-3424465a9e78 admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:45:49.876733 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-167adbf9-96fb-489e-9fea-3424465a9e78 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:45:49.877019 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-167adbf9-96fb-489e-9fea-3424465a9e78 admin admin] PATH_INFO: `/v3/projects/e4c232c36277491b8fced135ddbedf73/users/ce40ff6198894a1ea06c6bbf155ab584/roles/8f6cc574056c4278bc9778f14dc4b411` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:45:49.895268 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-167adbf9-96fb-489e-9fea-3424465a9e78 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e4c232c36277491b8fced135ddbedf73, user_id=ce40ff6198894a1ea06c6bbf155ab584, role_id=8f6cc574056c4278bc9778f14dc4b411)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:45:49.898022 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-167adbf9-96fb-489e-9fea-3424465a9e78 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:45:49.913308 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1269/2513] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:45:49 2022] PUT /identity/v3/projects/e4c232c36277491b8fced135ddbedf73/users/ce40ff6198894a1ea06c6bbf155ab584/roles/8f6cc574056c4278bc9778f14dc4b411 => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:45:49.926779 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77196804-23c1-45ce-ac29-6e6d3cf54eca None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:45:49.927069 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77196804-23c1-45ce-ac29-6e6d3cf54eca None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:45:49.927349 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77196804-23c1-45ce-ac29-6e6d3cf54eca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:45:49.963545 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-77196804-23c1-45ce-ac29-6e6d3cf54eca None None] MFA Rules not processed for user `ce40ff6198894a1ea06c6bbf155ab584`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:45:49.965649 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-77196804-23c1-45ce-ac29-6e6d3cf54eca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:45:49.977078 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:45:50.045364 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1245/2514] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:45:49 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:45:50.059714 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-167adbf9-96fb-489e-9fea-3424465a9e78 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:45:50.061872 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-167adbf9-96fb-489e-9fea-3424465a9e78 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:45:50.142157 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-167adbf9-96fb-489e-9fea-3424465a9e78 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:45:50.146691 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0db63b42-3256-4bf8-a8ce-43f318dcf271 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ffbc136c77b84e8c87128f7247f4ec03', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:45:50.147264 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db63b42-3256-4bf8-a8ce-43f318dcf271 service cinder] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:45:50.147548 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db63b42-3256-4bf8-a8ce-43f318dcf271 service cinder] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:45:50.147826 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db63b42-3256-4bf8-a8ce-43f318dcf271 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:45:50.159466 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0db63b42-3256-4bf8-a8ce-43f318dcf271 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:45:50.161045 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0db63b42-3256-4bf8-a8ce-43f318dcf271 service cinder] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:45:50.174415 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1270/2515] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:45:50 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 115 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:45:57.816692 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77196804-23c1-45ce-ac29-6e6d3cf54eca None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:45:57.818103 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-77196804-23c1-45ce-ac29-6e6d3cf54eca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:45:57.875045 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77196804-23c1-45ce-ac29-6e6d3cf54eca None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:45:57.879058 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-482d528d-7d65-499c-8924-bbd10d2f3cce service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:45:57.879659 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-482d528d-7d65-499c-8924-bbd10d2f3cce service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:45:57.879933 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-482d528d-7d65-499c-8924-bbd10d2f3cce service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:45:57.880197 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-482d528d-7d65-499c-8924-bbd10d2f3cce service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:45:57.881928 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-482d528d-7d65-499c-8924-bbd10d2f3cce service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:45:57.965051 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-482d528d-7d65-499c-8924-bbd10d2f3cce service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:45:57.966502 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-482d528d-7d65-499c-8924-bbd10d2f3cce service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:45:57.986793 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1246/2516] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:45:57 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 171 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:45:59.176407 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0db63b42-3256-4bf8-a8ce-43f318dcf271 service cinder] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:45:59.185797 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0db63b42-3256-4bf8-a8ce-43f318dcf271 service cinder] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:45:59.189430 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32f78289-ebf7-4a5e-8efc-42cde28ac3bd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:45:59.190018 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32f78289-ebf7-4a5e-8efc-42cde28ac3bd service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:45:59.190299 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32f78289-ebf7-4a5e-8efc-42cde28ac3bd service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:45:59.190591 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32f78289-ebf7-4a5e-8efc-42cde28ac3bd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:45:59.192287 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-32f78289-ebf7-4a5e-8efc-42cde28ac3bd service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:45:59.205609 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32f78289-ebf7-4a5e-8efc-42cde28ac3bd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:45:59.207044 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32f78289-ebf7-4a5e-8efc-42cde28ac3bd service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:45:59.219100 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1271/2517] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:45:59 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:46:03.886918 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-482d528d-7d65-499c-8924-bbd10d2f3cce service neutron] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:46:03.888830 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-482d528d-7d65-499c-8924-bbd10d2f3cce service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:46:03.903165 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-482d528d-7d65-499c-8924-bbd10d2f3cce service neutron] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:46:03.909115 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51414ec9-3765-4cc0-856b-ad6f0288309f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e703b42c53974daaa608b0ccc3f018b1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'ResellerAdmin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:46:03.909893 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51414ec9-3765-4cc0-856b-ad6f0288309f service nova] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:46:03.910189 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51414ec9-3765-4cc0-856b-ad6f0288309f service nova] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:46:03.910459 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51414ec9-3765-4cc0-856b-ad6f0288309f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:46:03.912196 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-51414ec9-3765-4cc0-856b-ad6f0288309f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:46:03.946842 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:46:03.996883 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51414ec9-3765-4cc0-856b-ad6f0288309f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:46:03.998449 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51414ec9-3765-4cc0-856b-ad6f0288309f service nova] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:46:04.018681 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1247/2518] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:46:03 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 132 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:46:09.213089 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=57,keystone:INSERT=2 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:46:10.032032 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d13eea0-f26c-440f-a279-793e06705116 None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:46:10.032316 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d13eea0-f26c-440f-a279-793e06705116 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:46:10.032588 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d13eea0-f26c-440f-a279-793e06705116 None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:46:10.033962 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1272/2519] 10.222.0.25 () {54 vars in 921 bytes} [Sun Sep 25 04:46:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 25 04:46:10.040183 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1735ff7-fc59-49d6-b890-aea8093784b6 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:46:10.040741 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1735ff7-fc59-49d6-b890-aea8093784b6 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:46:10.040855 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1735ff7-fc59-49d6-b890-aea8093784b6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:46:10.081163 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-a1735ff7-fc59-49d6-b890-aea8093784b6 None None] MFA Rules not processed for user `e703b42c53974daaa608b0ccc3f018b1`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:46:10.082394 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-a1735ff7-fc59-49d6-b890-aea8093784b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:46:10.096669 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1248/2520] 10.222.0.25 () {58 vars in 1018 bytes} [Sun Sep 25 04:46:10 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 57 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:46:10.108099 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d13eea0-f26c-440f-a279-793e06705116 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:46:10.109970 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-8d13eea0-f26c-440f-a279-793e06705116 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:46:10.221502 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d13eea0-f26c-440f-a279-793e06705116 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:46:10.225313 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9db3181d-1d1d-4464-91a0-ce27b3c54dd5 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '316eaa67d5464091af47efd489b96b87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:46:10.225929 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db3181d-1d1d-4464-91a0-ce27b3c54dd5 service glance] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:46:10.226209 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db3181d-1d1d-4464-91a0-ce27b3c54dd5 service glance] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:46:10.226479 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db3181d-1d1d-4464-91a0-ce27b3c54dd5 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:46:10.236840 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9db3181d-1d1d-4464-91a0-ce27b3c54dd5 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:46:10.238325 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9db3181d-1d1d-4464-91a0-ce27b3c54dd5 service glance] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:46:10.250056 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1273/2521] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:46:10 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 143 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 25 04:46:10.320343 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be9f0a7b-fdc7-4e26-8a9b-8c86b0dbe7c2 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:46:10.320628 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be9f0a7b-fdc7-4e26-8a9b-8c86b0dbe7c2 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:46:10.320933 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be9f0a7b-fdc7-4e26-8a9b-8c86b0dbe7c2 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:46:10.322310 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1249/2522] 10.222.0.25 () {54 vars in 921 bytes} [Sun Sep 25 04:46:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:46:10.327906 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce216ffe-c7da-4ff0-9c34-b0c31851cfe4 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:46:10.328197 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce216ffe-c7da-4ff0-9c34-b0c31851cfe4 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:46:10.328472 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce216ffe-c7da-4ff0-9c34-b0c31851cfe4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:46:10.371303 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-ce216ffe-c7da-4ff0-9c34-b0c31851cfe4 None None] MFA Rules not processed for user `0268e88e0c5645a98bab5664be6e49bc`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:46:10.372460 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-ce216ffe-c7da-4ff0-9c34-b0c31851cfe4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:46:10.380011 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1274/2523] 10.222.0.25 () {58 vars in 1018 bytes} [Sun Sep 25 04:46:10 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 54 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:46:10.393155 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be9f0a7b-fdc7-4e26-8a9b-8c86b0dbe7c2 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:46:10.395093 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-be9f0a7b-fdc7-4e26-8a9b-8c86b0dbe7c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:46:10.408772 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be9f0a7b-fdc7-4e26-8a9b-8c86b0dbe7c2 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:46:10.412301 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-329df516-69b1-4b74-bbc4-d7c3aeaf5ccd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:46:10.412923 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-329df516-69b1-4b74-bbc4-d7c3aeaf5ccd service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:46:10.413203 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-329df516-69b1-4b74-bbc4-d7c3aeaf5ccd service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:46:10.413473 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-329df516-69b1-4b74-bbc4-d7c3aeaf5ccd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:46:10.420761 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-329df516-69b1-4b74-bbc4-d7c3aeaf5ccd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:46:10.422190 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-329df516-69b1-4b74-bbc4-d7c3aeaf5ccd service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:46:10.431708 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1250/2524] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:46:10 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:46:10.520130 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-518dfd01-0777-422c-89fb-1051bee396ef None None] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:46:10.520409 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-518dfd01-0777-422c-89fb-1051bee396ef None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:46:10.520700 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-518dfd01-0777-422c-89fb-1051bee396ef None None] PATH_INFO: `/` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:46:10.522086 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1275/2525] 10.222.0.25 () {54 vars in 921 bytes} [Sun Sep 25 04:46:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 25 04:46:10.527776 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d03a5c74-6298-4fe5-979e-01272911072e None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:46:10.528062 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d03a5c74-6298-4fe5-979e-01272911072e None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:46:10.528336 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d03a5c74-6298-4fe5-979e-01272911072e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:46:10.549606 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-d03a5c74-6298-4fe5-979e-01272911072e None None] MFA Rules not processed for user `e703b42c53974daaa608b0ccc3f018b1`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:46:10.550855 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-d03a5c74-6298-4fe5-979e-01272911072e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:46:10.560074 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1251/2526] 10.222.0.25 () {58 vars in 1018 bytes} [Sun Sep 25 04:46:10 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:46:10.571259 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-518dfd01-0777-422c-89fb-1051bee396ef None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:46:10.583090 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-518dfd01-0777-422c-89fb-1051bee396ef None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:46:10.586650 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad932c97-3b39-4180-aab5-17d17e0036e8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:46:10.587240 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad932c97-3b39-4180-aab5-17d17e0036e8 service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:46:10.587517 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad932c97-3b39-4180-aab5-17d17e0036e8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:46:10.587785 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad932c97-3b39-4180-aab5-17d17e0036e8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:46:10.598299 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad932c97-3b39-4180-aab5-17d17e0036e8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:46:10.599721 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad932c97-3b39-4180-aab5-17d17e0036e8 service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:46:10.611165 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1276/2527] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:46:10 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:46:12.631712 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-514430e8-d583-408f-b556-801f4404fd90 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:46:12.632178 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-514430e8-d583-408f-b556-801f4404fd90 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:46:12.632272 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-514430e8-d583-408f-b556-801f4404fd90 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:46:12.686945 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-514430e8-d583-408f-b556-801f4404fd90 None None] MFA Rules not processed for user `050b3aab44bb4b17aa4a1ba7cccdb231`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:46:12.688170 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-514430e8-d583-408f-b556-801f4404fd90 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:46:12.761326 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1252/2528] 10.222.0.25 () {58 vars in 1009 bytes} [Sun Sep 25 04:46:12 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:46:12.773603 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad932c97-3b39-4180-aab5-17d17e0036e8 service neutron] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:46:12.775021 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-ad932c97-3b39-4180-aab5-17d17e0036e8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:46:12.869241 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad932c97-3b39-4180-aab5-17d17e0036e8 service neutron] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:46:12.874990 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-902b5e85-a020-4e01-bdf0-c3c6523f6890 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '23ea88f46e7740d295148a93ade29dd9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:46:12.875565 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-902b5e85-a020-4e01-bdf0-c3c6523f6890 service swift] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:46:12.875929 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-902b5e85-a020-4e01-bdf0-c3c6523f6890 service swift] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:46:12.876216 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-902b5e85-a020-4e01-bdf0-c3c6523f6890 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:46:12.885116 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-902b5e85-a020-4e01-bdf0-c3c6523f6890 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:46:12.886712 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-902b5e85-a020-4e01-bdf0-c3c6523f6890 service swift] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:46:12.895668 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1277/2529] 10.222.0.25 () {60 vars in 1384 bytes} [Sun Sep 25 04:46:12 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 123 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 25 04:46:22.758422 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=84 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:46:22.892389 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=91 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:46:23.587723 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-514430e8-d583-408f-b556-801f4404fd90 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:46:23.598129 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-514430e8-d583-408f-b556-801f4404fd90 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:46:23.602304 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0e22ed2-d7a1-4436-a9c8-dd289fc4b6c9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ffbc136c77b84e8c87128f7247f4ec03', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:46:23.602984 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0e22ed2-d7a1-4436-a9c8-dd289fc4b6c9 service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:46:23.603353 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0e22ed2-d7a1-4436-a9c8-dd289fc4b6c9 service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:46:23.603686 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0e22ed2-d7a1-4436-a9c8-dd289fc4b6c9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:46:23.611535 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-a0e22ed2-d7a1-4436-a9c8-dd289fc4b6c9 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:46:23.697939 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0e22ed2-d7a1-4436-a9c8-dd289fc4b6c9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:46:23.699455 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0e22ed2-d7a1-4436-a9c8-dd289fc4b6c9 service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:46:23.724591 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1253/2530] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:46:23 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 137 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:46:33.721591 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=41 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:46:42.668415 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-902b5e85-a020-4e01-bdf0-c3c6523f6890 service swift] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:46:42.745652 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-902b5e85-a020-4e01-bdf0-c3c6523f6890 service swift] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:46:42.751270 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cae8290-39cb-41c6-81e2-16aa84079332 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:46:42.751946 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cae8290-39cb-41c6-81e2-16aa84079332 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:46:42.752226 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cae8290-39cb-41c6-81e2-16aa84079332 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:46:42.752495 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cae8290-39cb-41c6-81e2-16aa84079332 admin admin] PATH_INFO: `/v3/users/ce40ff6198894a1ea06c6bbf155ab584` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:46:42.753674 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cae8290-39cb-41c6-81e2-16aa84079332 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ce40ff6198894a1ea06c6bbf155ab584)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:46:42.755595 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cae8290-39cb-41c6-81e2-16aa84079332 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:46:42.805360 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-6cae8290-39cb-41c6-81e2-16aa84079332 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ce40ff6198894a1ea06c6bbf155ab584'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:46:42.814348 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-6cae8290-39cb-41c6-81e2-16aa84079332 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ce40ff6198894a1ea06c6bbf155ab584'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:46:42.832843 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-6cae8290-39cb-41c6-81e2-16aa84079332 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ce40ff6198894a1ea06c6bbf155ab584'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:46:42.833989 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-6cae8290-39cb-41c6-81e2-16aa84079332 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ce40ff6198894a1ea06c6bbf155ab584'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:46:42.843887 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-6cae8290-39cb-41c6-81e2-16aa84079332 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ce40ff6198894a1ea06c6bbf155ab584'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:46:42.850347 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1278/2531] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:46:42 2022] DELETE /identity/v3/users/ce40ff6198894a1ea06c6bbf155ab584 => generated 0 bytes in 182 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:46:42.854857 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ecbd60d-42cc-48cf-86cd-0f50e3de2ce5 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:46:42.855354 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ecbd60d-42cc-48cf-86cd-0f50e3de2ce5 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:46:42.855553 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ecbd60d-42cc-48cf-86cd-0f50e3de2ce5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:46:42.878456 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-7ecbd60d-42cc-48cf-86cd-0f50e3de2ce5 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:46:42.879613 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-7ecbd60d-42cc-48cf-86cd-0f50e3de2ce5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:46:42.942936 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1254/2532] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:46:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:46:42.952734 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cae8290-39cb-41c6-81e2-16aa84079332 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:46:42.954163 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-6cae8290-39cb-41c6-81e2-16aa84079332 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:46:43.011832 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cae8290-39cb-41c6-81e2-16aa84079332 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:46:43.015708 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b22e826-6a81-4adb-b58f-cf9addfee16f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:46:43.016332 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b22e826-6a81-4adb-b58f-cf9addfee16f service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:46:43.016615 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b22e826-6a81-4adb-b58f-cf9addfee16f service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:46:43.016925 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b22e826-6a81-4adb-b58f-cf9addfee16f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:46:43.026253 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b22e826-6a81-4adb-b58f-cf9addfee16f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:46:43.027689 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b22e826-6a81-4adb-b58f-cf9addfee16f service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:46:43.038079 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1279/2533] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:46:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 25 04:46:43.082536 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ecbd60d-42cc-48cf-86cd-0f50e3de2ce5 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:46:43.084336 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-7ecbd60d-42cc-48cf-86cd-0f50e3de2ce5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:46:43.097897 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ecbd60d-42cc-48cf-86cd-0f50e3de2ce5 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:46:43.101812 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cd7ea72-dbf4-4bf4-9a88-79a229bca70c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:46:43.102475 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cd7ea72-dbf4-4bf4-9a88-79a229bca70c service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:46:43.102755 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cd7ea72-dbf4-4bf4-9a88-79a229bca70c service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:46:43.103166 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cd7ea72-dbf4-4bf4-9a88-79a229bca70c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:46:43.104906 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-3cd7ea72-dbf4-4bf4-9a88-79a229bca70c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:46:43.116104 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cd7ea72-dbf4-4bf4-9a88-79a229bca70c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:46:43.117767 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cd7ea72-dbf4-4bf4-9a88-79a229bca70c service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:46:43.127009 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1255/2534] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:46:43 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:46:43.309202 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b22e826-6a81-4adb-b58f-cf9addfee16f service neutron] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:46:43.310785 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-2b22e826-6a81-4adb-b58f-cf9addfee16f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:46:43.329743 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b22e826-6a81-4adb-b58f-cf9addfee16f service neutron] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:46:43.334787 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cfdd0df-1393-4814-b986-8069459d8c0a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:46:43.335294 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cfdd0df-1393-4814-b986-8069459d8c0a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:46:43.335576 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cfdd0df-1393-4814-b986-8069459d8c0a admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:46:43.335852 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cfdd0df-1393-4814-b986-8069459d8c0a admin admin] PATH_INFO: `/v3/projects/e4c232c36277491b8fced135ddbedf73` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:46:43.337011 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cfdd0df-1393-4814-b986-8069459d8c0a admin admin] RBAC: Authorizing `identity:delete_project(project_id=e4c232c36277491b8fced135ddbedf73)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:46:43.338930 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cfdd0df-1393-4814-b986-8069459d8c0a admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:46:43.358141 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:46:43.392256 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-2cfdd0df-1393-4814-b986-8069459d8c0a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e4c232c36277491b8fced135ddbedf73', 'request_id': 'req-2cfdd0df-1393-4814-b986-8069459d8c0a'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:46:43.400509 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-2cfdd0df-1393-4814-b986-8069459d8c0a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e4c232c36277491b8fced135ddbedf73', 'request_id': 'req-2cfdd0df-1393-4814-b986-8069459d8c0a'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:46:43.407220 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-2cfdd0df-1393-4814-b986-8069459d8c0a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e4c232c36277491b8fced135ddbedf73', 'request_id': 'req-2cfdd0df-1393-4814-b986-8069459d8c0a'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:46:43.419370 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-2cfdd0df-1393-4814-b986-8069459d8c0a admin admin] The token cache is being invalidate because project e4c232c36277491b8fced135ddbedf73 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81434) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:46:43.419690 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-2cfdd0df-1393-4814-b986-8069459d8c0a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:46:43.421171 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-2cfdd0df-1393-4814-b986-8069459d8c0a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:46:43.423197 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1280/2535] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:46:43 2022] DELETE /identity/v3/projects/e4c232c36277491b8fced135ddbedf73 => generated 0 bytes in 114 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:46:45.604556 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-241ad14c-765f-456b-b5d7-e7e3fc2a282c None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:46:45.605051 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-241ad14c-765f-456b-b5d7-e7e3fc2a282c None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:46:45.605270 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-241ad14c-765f-456b-b5d7-e7e3fc2a282c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:46:45.626094 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-241ad14c-765f-456b-b5d7-e7e3fc2a282c None None] MFA Rules not processed for user `050b3aab44bb4b17aa4a1ba7cccdb231`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:46:45.627296 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-241ad14c-765f-456b-b5d7-e7e3fc2a282c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:46:45.684139 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1256/2536] 10.222.0.25 () {58 vars in 967 bytes} [Sun Sep 25 04:46:45 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:46:45.699188 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cfdd0df-1393-4814-b986-8069459d8c0a admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:46:45.700686 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-2cfdd0df-1393-4814-b986-8069459d8c0a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:46:45.776227 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cfdd0df-1393-4814-b986-8069459d8c0a admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:46:45.781912 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adbca0b6-9049-4709-9398-a1e04b92f055 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '23ea88f46e7740d295148a93ade29dd9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:46:45.782476 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adbca0b6-9049-4709-9398-a1e04b92f055 service swift] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:46:45.782780 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adbca0b6-9049-4709-9398-a1e04b92f055 service swift] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:46:45.783012 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adbca0b6-9049-4709-9398-a1e04b92f055 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:46:45.791546 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adbca0b6-9049-4709-9398-a1e04b92f055 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:46:45.793161 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adbca0b6-9049-4709-9398-a1e04b92f055 service swift] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:46:45.802104 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1281/2537] 10.222.0.25 () {60 vars in 1384 bytes} [Sun Sep 25 04:46:45 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 25 04:46:50.705360 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-241ad14c-765f-456b-b5d7-e7e3fc2a282c None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:46:50.707184 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-241ad14c-765f-456b-b5d7-e7e3fc2a282c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:46:50.820031 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-241ad14c-765f-456b-b5d7-e7e3fc2a282c None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:46:50.826577 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cb9aea5-849d-473c-b63a-6adf6ade1cb5 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e703b42c53974daaa608b0ccc3f018b1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'ResellerAdmin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:46:50.827271 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cb9aea5-849d-473c-b63a-6adf6ade1cb5 service nova] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:46:50.828332 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cb9aea5-849d-473c-b63a-6adf6ade1cb5 service nova] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:46:50.828824 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cb9aea5-849d-473c-b63a-6adf6ade1cb5 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:46:50.831334 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-7cb9aea5-849d-473c-b63a-6adf6ade1cb5 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:46:50.865986 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cb9aea5-849d-473c-b63a-6adf6ade1cb5 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:46:50.867587 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cb9aea5-849d-473c-b63a-6adf6ade1cb5 service nova] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:46:50.889208 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1257/2538] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:46:50 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 184 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:46:55.798703 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=49,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:46:56.313579 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adbca0b6-9049-4709-9398-a1e04b92f055 service swift] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:46:56.315730 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-adbca0b6-9049-4709-9398-a1e04b92f055 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:46:56.380578 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adbca0b6-9049-4709-9398-a1e04b92f055 service swift] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:46:56.384163 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af24ca81-eb86-4263-b108-9f9f6d6d448b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:46:56.384802 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af24ca81-eb86-4263-b108-9f9f6d6d448b service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:46:56.385116 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af24ca81-eb86-4263-b108-9f9f6d6d448b service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:46:56.385486 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af24ca81-eb86-4263-b108-9f9f6d6d448b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:46:56.387221 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-af24ca81-eb86-4263-b108-9f9f6d6d448b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:46:56.449846 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af24ca81-eb86-4263-b108-9f9f6d6d448b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:46:56.451293 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af24ca81-eb86-4263-b108-9f9f6d6d448b service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:46:56.460961 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1282/2539] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:46:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 149 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:46:59.705151 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cb9aea5-849d-473c-b63a-6adf6ade1cb5 service nova] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:46:59.706640 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-7cb9aea5-849d-473c-b63a-6adf6ade1cb5 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:46:59.720564 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cb9aea5-849d-473c-b63a-6adf6ade1cb5 service nova] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:46:59.725816 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8da9784e-762d-47b7-98e6-c53a80dd6f05 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:46:59.726461 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8da9784e-762d-47b7-98e6-c53a80dd6f05 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:46:59.726745 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8da9784e-762d-47b7-98e6-c53a80dd6f05 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:46:59.727023 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8da9784e-762d-47b7-98e6-c53a80dd6f05 admin admin] PATH_INFO: `/v3/users/16281c79f73e4063b52066966c393148` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:46:59.728198 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8da9784e-762d-47b7-98e6-c53a80dd6f05 admin admin] RBAC: Authorizing `identity:delete_user(user_id=16281c79f73e4063b52066966c393148)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:46:59.730166 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8da9784e-762d-47b7-98e6-c53a80dd6f05 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:46:59.745448 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:DELETE=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:46:59.785423 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-8da9784e-762d-47b7-98e6-c53a80dd6f05 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '16281c79f73e4063b52066966c393148'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:46:59.793391 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-8da9784e-762d-47b7-98e6-c53a80dd6f05 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '16281c79f73e4063b52066966c393148'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:46:59.806580 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-8da9784e-762d-47b7-98e6-c53a80dd6f05 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '16281c79f73e4063b52066966c393148'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:46:59.807299 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-8da9784e-762d-47b7-98e6-c53a80dd6f05 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '16281c79f73e4063b52066966c393148'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:46:59.815254 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-8da9784e-762d-47b7-98e6-c53a80dd6f05 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '16281c79f73e4063b52066966c393148'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:46:59.821426 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1258/2540] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:46:59 2022] DELETE /identity/v3/users/16281c79f73e4063b52066966c393148 => generated 0 bytes in 117 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:46:59.993874 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af24ca81-eb86-4263-b108-9f9f6d6d448b service neutron] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:00.005351 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af24ca81-eb86-4263-b108-9f9f6d6d448b service neutron] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:00.063734 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7dc5908-201a-4775-843d-464ba4684063 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:00.064378 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7dc5908-201a-4775-843d-464ba4684063 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:00.064689 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7dc5908-201a-4775-843d-464ba4684063 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:00.064967 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7dc5908-201a-4775-843d-464ba4684063 admin admin] PATH_INFO: `/v3/projects/e1108f4c9da246cb9b4b9f89833ceeda` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:00.066084 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7dc5908-201a-4775-843d-464ba4684063 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e1108f4c9da246cb9b4b9f89833ceeda)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:00.068020 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7dc5908-201a-4775-843d-464ba4684063 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:00.115774 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-b7dc5908-201a-4775-843d-464ba4684063 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e1108f4c9da246cb9b4b9f89833ceeda', 'request_id': 'req-b7dc5908-201a-4775-843d-464ba4684063'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:00.118168 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=95,keystone:DELETE=4,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:47:00.124768 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-b7dc5908-201a-4775-843d-464ba4684063 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e1108f4c9da246cb9b4b9f89833ceeda', 'request_id': 'req-b7dc5908-201a-4775-843d-464ba4684063'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:00.131583 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-b7dc5908-201a-4775-843d-464ba4684063 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e1108f4c9da246cb9b4b9f89833ceeda', 'request_id': 'req-b7dc5908-201a-4775-843d-464ba4684063'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:00.142633 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-b7dc5908-201a-4775-843d-464ba4684063 admin admin] The token cache is being invalidate because project e1108f4c9da246cb9b4b9f89833ceeda was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81434) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:47:00.142926 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-b7dc5908-201a-4775-843d-464ba4684063 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:00.143559 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-b7dc5908-201a-4775-843d-464ba4684063 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:00.145096 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1283/2541] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:46:59 2022] DELETE /identity/v3/projects/e1108f4c9da246cb9b4b9f89833ceeda => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:47:00.150248 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b7b8250-46c6-490f-a33b-4e5cf33c93d3 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:00.150541 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b7b8250-46c6-490f-a33b-4e5cf33c93d3 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:00.150823 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b7b8250-46c6-490f-a33b-4e5cf33c93d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:00.172136 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-0b7b8250-46c6-490f-a33b-4e5cf33c93d3 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:47:00.173453 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-0b7b8250-46c6-490f-a33b-4e5cf33c93d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:47:00.232949 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1259/2542] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:47:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 04:47:00.242823 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46742086-0d0e-463d-8f41-37c702874473 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:00.243049 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46742086-0d0e-463d-8f41-37c702874473 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:00.243325 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46742086-0d0e-463d-8f41-37c702874473 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:00.264805 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-46742086-0d0e-463d-8f41-37c702874473 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:47:00.266096 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-46742086-0d0e-463d-8f41-37c702874473 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:47:00.274421 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1284/2543] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:47:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:47:00.278687 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b7b8250-46c6-490f-a33b-4e5cf33c93d3 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:00.288899 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b7b8250-46c6-490f-a33b-4e5cf33c93d3 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:00.293623 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd59c93f-8912-438c-844a-32524e6117e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:00.294223 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd59c93f-8912-438c-844a-32524e6117e8 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:00.294507 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd59c93f-8912-438c-844a-32524e6117e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:00.294784 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd59c93f-8912-438c-844a-32524e6117e8 admin admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:00.295431 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd59c93f-8912-438c-844a-32524e6117e8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:47:00.296238 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd59c93f-8912-438c-844a-32524e6117e8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:00.297906 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd59c93f-8912-438c-844a-32524e6117e8 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:00.309185 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1260/2544] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Sep 25 04:47:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:47:00.314232 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46742086-0d0e-463d-8f41-37c702874473 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:00.323295 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46742086-0d0e-463d-8f41-37c702874473 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:00.327930 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ada70748-e374-465c-8bfb-708854316bcb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:00.328512 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ada70748-e374-465c-8bfb-708854316bcb admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:00.328821 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ada70748-e374-465c-8bfb-708854316bcb admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:00.329091 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ada70748-e374-465c-8bfb-708854316bcb admin admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:00.329708 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ada70748-e374-465c-8bfb-708854316bcb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:00.331567 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ada70748-e374-465c-8bfb-708854316bcb admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:00.362042 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1285/2545] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:47:00 2022] POST /identity/v3/projects => generated 385 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 04:47:00.366323 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd59c93f-8912-438c-844a-32524e6117e8 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:00.377673 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd59c93f-8912-438c-844a-32524e6117e8 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:00.433875 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5cf15d2-ae2e-42b7-994a-6600d01d937d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:00.434482 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5cf15d2-ae2e-42b7-994a-6600d01d937d admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:00.434763 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5cf15d2-ae2e-42b7-994a-6600d01d937d admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:00.435036 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5cf15d2-ae2e-42b7-994a-6600d01d937d admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:00.435670 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5cf15d2-ae2e-42b7-994a-6600d01d937d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:00.438352 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5cf15d2-ae2e-42b7-994a-6600d01d937d admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:00.458892 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1261/2546] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:47:00 2022] POST /identity/v3/users => generated 293 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:47:00.463148 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ada70748-e374-465c-8bfb-708854316bcb admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:00.474691 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ada70748-e374-465c-8bfb-708854316bcb admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:00.479417 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fa1abfe-441c-41d7-ae64-65f0c41edb73 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:00.480034 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fa1abfe-441c-41d7-ae64-65f0c41edb73 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:00.480315 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fa1abfe-441c-41d7-ae64-65f0c41edb73 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:00.480584 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fa1abfe-441c-41d7-ae64-65f0c41edb73 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:00.481252 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fa1abfe-441c-41d7-ae64-65f0c41edb73 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:00.482843 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fa1abfe-441c-41d7-ae64-65f0c41edb73 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:00.494115 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1286/2547] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:47:00 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:47:00.498140 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5cf15d2-ae2e-42b7-994a-6600d01d937d admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:00.508485 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5cf15d2-ae2e-42b7-994a-6600d01d937d admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:00.513031 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b1678e7-96c3-469b-89e0-ce28bed5bdd8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:00.513653 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b1678e7-96c3-469b-89e0-ce28bed5bdd8 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:00.513928 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b1678e7-96c3-469b-89e0-ce28bed5bdd8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:00.514198 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b1678e7-96c3-469b-89e0-ce28bed5bdd8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:00.515034 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b1678e7-96c3-469b-89e0-ce28bed5bdd8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:00.516622 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b1678e7-96c3-469b-89e0-ce28bed5bdd8 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:00.517880 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=87,keystone:DELETE=9,keystone:INSERT=4 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:47:00.529135 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1262/2548] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:47:00 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:47:00.533679 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fa1abfe-441c-41d7-ae64-65f0c41edb73 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:00.543615 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fa1abfe-441c-41d7-ae64-65f0c41edb73 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:00.549038 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df402fe4-1a6a-4214-a2ae-cfe4bccb204b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:00.549679 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df402fe4-1a6a-4214-a2ae-cfe4bccb204b admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:00.549957 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df402fe4-1a6a-4214-a2ae-cfe4bccb204b admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:00.550229 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df402fe4-1a6a-4214-a2ae-cfe4bccb204b admin admin] PATH_INFO: `/v3/projects/51b6143a4c7549d3b0abfabff6e2dbac/users/ae7cd2182632480c832e52790cce7095/roles/8f6cc574056c4278bc9778f14dc4b411` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:00.564495 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df402fe4-1a6a-4214-a2ae-cfe4bccb204b admin admin] RBAC: Authorizing `identity:create_grant(project_id=51b6143a4c7549d3b0abfabff6e2dbac, user_id=ae7cd2182632480c832e52790cce7095, role_id=8f6cc574056c4278bc9778f14dc4b411)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:00.567146 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df402fe4-1a6a-4214-a2ae-cfe4bccb204b admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:00.579456 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1287/2549] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:47:00 2022] PUT /identity/v3/projects/51b6143a4c7549d3b0abfabff6e2dbac/users/ae7cd2182632480c832e52790cce7095/roles/8f6cc574056c4278bc9778f14dc4b411 => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 04:47:00.590097 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5703c6e-3279-4489-acf4-b6a81462d6fe None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:00.590379 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5703c6e-3279-4489-acf4-b6a81462d6fe None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:00.590654 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5703c6e-3279-4489-acf4-b6a81462d6fe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:00.619698 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-d5703c6e-3279-4489-acf4-b6a81462d6fe None None] MFA Rules not processed for user `ae7cd2182632480c832e52790cce7095`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:47:00.620929 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-d5703c6e-3279-4489-acf4-b6a81462d6fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:47:00.684875 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1263/2550] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:47:00 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:47:00.696007 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df402fe4-1a6a-4214-a2ae-cfe4bccb204b admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:00.697740 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-df402fe4-1a6a-4214-a2ae-cfe4bccb204b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:47:00.772501 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df402fe4-1a6a-4214-a2ae-cfe4bccb204b admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:00.777949 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c185276a-4539-42d8-b25a-da5afe5f6daf service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e703b42c53974daaa608b0ccc3f018b1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'ResellerAdmin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:00.778556 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c185276a-4539-42d8-b25a-da5afe5f6daf service nova] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:00.778835 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c185276a-4539-42d8-b25a-da5afe5f6daf service nova] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:00.779104 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c185276a-4539-42d8-b25a-da5afe5f6daf service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:00.787612 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c185276a-4539-42d8-b25a-da5afe5f6daf service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:00.789291 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c185276a-4539-42d8-b25a-da5afe5f6daf service nova] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:00.799014 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1288/2551] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:47:00 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:47:10.168688 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5703c6e-3279-4489-acf4-b6a81462d6fe None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:10.229070 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5703c6e-3279-4489-acf4-b6a81462d6fe None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:10.233626 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d9a7a2d-cdaf-4fb6-a182-ca67eb8d7934 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:10.234261 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d9a7a2d-cdaf-4fb6-a182-ca67eb8d7934 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:10.234545 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d9a7a2d-cdaf-4fb6-a182-ca67eb8d7934 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:10.235385 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d9a7a2d-cdaf-4fb6-a182-ca67eb8d7934 admin admin] PATH_INFO: `/v3/users/ae7cd2182632480c832e52790cce7095` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:10.236901 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d9a7a2d-cdaf-4fb6-a182-ca67eb8d7934 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ae7cd2182632480c832e52790cce7095)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:10.239262 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d9a7a2d-cdaf-4fb6-a182-ca67eb8d7934 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:10.285336 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-3d9a7a2d-cdaf-4fb6-a182-ca67eb8d7934 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ae7cd2182632480c832e52790cce7095'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:10.292421 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-3d9a7a2d-cdaf-4fb6-a182-ca67eb8d7934 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ae7cd2182632480c832e52790cce7095'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:10.304795 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-3d9a7a2d-cdaf-4fb6-a182-ca67eb8d7934 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ae7cd2182632480c832e52790cce7095'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:10.305421 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-3d9a7a2d-cdaf-4fb6-a182-ca67eb8d7934 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ae7cd2182632480c832e52790cce7095'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:10.312188 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-3d9a7a2d-cdaf-4fb6-a182-ca67eb8d7934 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ae7cd2182632480c832e52790cce7095'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:10.318221 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1264/2552] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:47:10 2022] DELETE /identity/v3/users/ae7cd2182632480c832e52790cce7095 => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:47:10.322661 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88fe3fb3-37e2-4574-a560-466c9a814c16 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:10.323120 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88fe3fb3-37e2-4574-a560-466c9a814c16 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:10.323217 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88fe3fb3-37e2-4574-a560-466c9a814c16 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:10.342077 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-88fe3fb3-37e2-4574-a560-466c9a814c16 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:47:10.343284 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-88fe3fb3-37e2-4574-a560-466c9a814c16 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:47:10.404272 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1289/2553] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:47:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:47:10.414889 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d9a7a2d-cdaf-4fb6-a182-ca67eb8d7934 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:10.416307 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-3d9a7a2d-cdaf-4fb6-a182-ca67eb8d7934 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:47:10.423742 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:47:10.480146 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d9a7a2d-cdaf-4fb6-a182-ca67eb8d7934 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:10.483996 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e6b29c5-77fd-4eb6-a0c4-459b725c7d3f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:10.484598 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e6b29c5-77fd-4eb6-a0c4-459b725c7d3f service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:10.485079 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e6b29c5-77fd-4eb6-a0c4-459b725c7d3f service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:10.485572 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e6b29c5-77fd-4eb6-a0c4-459b725c7d3f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:10.495315 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e6b29c5-77fd-4eb6-a0c4-459b725c7d3f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:10.496782 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e6b29c5-77fd-4eb6-a0c4-459b725c7d3f service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:10.506863 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1265/2554] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:47:10 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:47:10.803171 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88fe3fb3-37e2-4574-a560-466c9a814c16 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:10.804664 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-88fe3fb3-37e2-4574-a560-466c9a814c16 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:47:10.813630 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:DELETE=3,keystone:SELECT=95,keystone:INSERT=2 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:47:10.819378 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88fe3fb3-37e2-4574-a560-466c9a814c16 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:10.825332 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f4a63ca-4b8a-43b9-be18-425b507a34b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:10.826188 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f4a63ca-4b8a-43b9-be18-425b507a34b2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:10.826905 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f4a63ca-4b8a-43b9-be18-425b507a34b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:10.827460 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f4a63ca-4b8a-43b9-be18-425b507a34b2 admin admin] PATH_INFO: `/v3/projects/51b6143a4c7549d3b0abfabff6e2dbac` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:10.829136 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f4a63ca-4b8a-43b9-be18-425b507a34b2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=51b6143a4c7549d3b0abfabff6e2dbac)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:10.831985 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f4a63ca-4b8a-43b9-be18-425b507a34b2 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:10.878715 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-9f4a63ca-4b8a-43b9-be18-425b507a34b2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '51b6143a4c7549d3b0abfabff6e2dbac', 'request_id': 'req-9f4a63ca-4b8a-43b9-be18-425b507a34b2'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:10.885371 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-9f4a63ca-4b8a-43b9-be18-425b507a34b2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '51b6143a4c7549d3b0abfabff6e2dbac', 'request_id': 'req-9f4a63ca-4b8a-43b9-be18-425b507a34b2'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:10.891202 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-9f4a63ca-4b8a-43b9-be18-425b507a34b2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '51b6143a4c7549d3b0abfabff6e2dbac', 'request_id': 'req-9f4a63ca-4b8a-43b9-be18-425b507a34b2'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:10.902330 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-9f4a63ca-4b8a-43b9-be18-425b507a34b2 admin admin] The token cache is being invalidate because project 51b6143a4c7549d3b0abfabff6e2dbac was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81434) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:47:10.902624 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-9f4a63ca-4b8a-43b9-be18-425b507a34b2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:10.903261 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-9f4a63ca-4b8a-43b9-be18-425b507a34b2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:10.904833 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1290/2555] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:47:10 2022] DELETE /identity/v3/projects/51b6143a4c7549d3b0abfabff6e2dbac => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:47:10.910278 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d018db59-6e69-48a2-ac41-b587df93d448 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:10.910565 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d018db59-6e69-48a2-ac41-b587df93d448 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:10.910841 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d018db59-6e69-48a2-ac41-b587df93d448 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:10.931819 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-d018db59-6e69-48a2-ac41-b587df93d448 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:47:10.933996 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-d018db59-6e69-48a2-ac41-b587df93d448 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:47:10.995655 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1266/2556] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:47:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:47:11.004766 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-314c5cc3-b344-411f-91c0-1396e56e8ef3 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:11.004994 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-314c5cc3-b344-411f-91c0-1396e56e8ef3 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:11.005269 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-314c5cc3-b344-411f-91c0-1396e56e8ef3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:11.025132 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-314c5cc3-b344-411f-91c0-1396e56e8ef3 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:47:11.026376 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-314c5cc3-b344-411f-91c0-1396e56e8ef3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:47:11.034885 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1291/2557] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:47:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 04:47:11.039363 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d018db59-6e69-48a2-ac41-b587df93d448 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:11.049777 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d018db59-6e69-48a2-ac41-b587df93d448 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:11.054433 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a50eed6-d1c7-4fcc-b1ce-950e7fb89e4b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:11.055026 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a50eed6-d1c7-4fcc-b1ce-950e7fb89e4b admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:11.055306 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a50eed6-d1c7-4fcc-b1ce-950e7fb89e4b admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:11.055577 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a50eed6-d1c7-4fcc-b1ce-950e7fb89e4b admin admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:11.056218 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a50eed6-d1c7-4fcc-b1ce-950e7fb89e4b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:47:11.056523 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a50eed6-d1c7-4fcc-b1ce-950e7fb89e4b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:11.058149 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a50eed6-d1c7-4fcc-b1ce-950e7fb89e4b admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:11.069600 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1267/2558] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Sep 25 04:47:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:47:11.073683 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-314c5cc3-b344-411f-91c0-1396e56e8ef3 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:11.083146 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-314c5cc3-b344-411f-91c0-1396e56e8ef3 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:11.088122 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ecc2725-71b0-412e-b7da-5830fb38bfbd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:11.088774 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ecc2725-71b0-412e-b7da-5830fb38bfbd admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:11.089079 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ecc2725-71b0-412e-b7da-5830fb38bfbd admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:11.089626 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ecc2725-71b0-412e-b7da-5830fb38bfbd admin admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:11.090262 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ecc2725-71b0-412e-b7da-5830fb38bfbd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:11.092164 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ecc2725-71b0-412e-b7da-5830fb38bfbd admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:11.121834 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1292/2559] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:47:11 2022] POST /identity/v3/projects => generated 381 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 04:47:11.126309 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a50eed6-d1c7-4fcc-b1ce-950e7fb89e4b admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:11.136296 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a50eed6-d1c7-4fcc-b1ce-950e7fb89e4b admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:11.193009 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-430ff43f-2713-4756-9b2c-e267517f76e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:11.193652 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-430ff43f-2713-4756-9b2c-e267517f76e6 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:11.193888 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-430ff43f-2713-4756-9b2c-e267517f76e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:11.194162 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-430ff43f-2713-4756-9b2c-e267517f76e6 admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:11.194918 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-430ff43f-2713-4756-9b2c-e267517f76e6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:11.196888 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-430ff43f-2713-4756-9b2c-e267517f76e6 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:11.218318 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1268/2560] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:47:11 2022] POST /identity/v3/users => generated 291 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:47:11.222533 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ecc2725-71b0-412e-b7da-5830fb38bfbd admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:11.232915 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ecc2725-71b0-412e-b7da-5830fb38bfbd admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:11.238478 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac235e8f-a412-4bd6-9af2-bc124ac811df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:11.239082 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac235e8f-a412-4bd6-9af2-bc124ac811df admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:11.239367 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac235e8f-a412-4bd6-9af2-bc124ac811df admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:11.239642 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac235e8f-a412-4bd6-9af2-bc124ac811df admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:11.240282 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac235e8f-a412-4bd6-9af2-bc124ac811df admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:11.241905 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac235e8f-a412-4bd6-9af2-bc124ac811df admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:11.252366 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1293/2561] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:47:11 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:47:11.256507 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-430ff43f-2713-4756-9b2c-e267517f76e6 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:11.267335 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-430ff43f-2713-4756-9b2c-e267517f76e6 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:11.271874 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-971a86cb-9b6f-4206-838c-b4a601b146ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:11.272465 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-971a86cb-9b6f-4206-838c-b4a601b146ab admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:11.272780 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-971a86cb-9b6f-4206-838c-b4a601b146ab admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:11.273050 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-971a86cb-9b6f-4206-838c-b4a601b146ab admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:11.273678 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-971a86cb-9b6f-4206-838c-b4a601b146ab admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:11.275269 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-971a86cb-9b6f-4206-838c-b4a601b146ab admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:11.285965 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1269/2562] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:47:11 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:47:11.289871 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac235e8f-a412-4bd6-9af2-bc124ac811df admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:11.300321 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac235e8f-a412-4bd6-9af2-bc124ac811df admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:11.305011 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b167005-2606-4520-994f-c8d354284421 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:11.305663 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b167005-2606-4520-994f-c8d354284421 admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:11.305944 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b167005-2606-4520-994f-c8d354284421 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:11.306220 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b167005-2606-4520-994f-c8d354284421 admin admin] PATH_INFO: `/v3/projects/340dd05680fc47f08828117ef16edf4f/users/1c406f7d87504c7d9ee67302b54d5409/roles/8f6cc574056c4278bc9778f14dc4b411` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:11.321047 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b167005-2606-4520-994f-c8d354284421 admin admin] RBAC: Authorizing `identity:create_grant(project_id=340dd05680fc47f08828117ef16edf4f, user_id=1c406f7d87504c7d9ee67302b54d5409, role_id=8f6cc574056c4278bc9778f14dc4b411)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:11.323686 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b167005-2606-4520-994f-c8d354284421 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:11.336426 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1294/2563] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:47:11 2022] PUT /identity/v3/projects/340dd05680fc47f08828117ef16edf4f/users/1c406f7d87504c7d9ee67302b54d5409/roles/8f6cc574056c4278bc9778f14dc4b411 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:47:11.346999 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a3222ee-ce9c-4d2a-8113-c1ca4c5c9200 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:11.347281 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a3222ee-ce9c-4d2a-8113-c1ca4c5c9200 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:11.347557 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a3222ee-ce9c-4d2a-8113-c1ca4c5c9200 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:11.372647 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:47:11.376628 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-9a3222ee-ce9c-4d2a-8113-c1ca4c5c9200 None None] MFA Rules not processed for user `1c406f7d87504c7d9ee67302b54d5409`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:47:11.378569 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-9a3222ee-ce9c-4d2a-8113-c1ca4c5c9200 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:47:11.444050 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1270/2564] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:47:11 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:47:11.448539 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b167005-2606-4520-994f-c8d354284421 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:11.512090 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b167005-2606-4520-994f-c8d354284421 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:11.514873 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c49ba992-c83e-4308-aa5e-4187b35fbee0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:11.517429 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c49ba992-c83e-4308-aa5e-4187b35fbee0 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:11.517429 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c49ba992-c83e-4308-aa5e-4187b35fbee0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:11.517429 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c49ba992-c83e-4308-aa5e-4187b35fbee0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:11.517429 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c49ba992-c83e-4308-aa5e-4187b35fbee0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:11.518730 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c49ba992-c83e-4308-aa5e-4187b35fbee0 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:11.544763 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1295/2565] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:47:11 2022] POST /identity/v3/projects => generated 381 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:47:11.549444 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a3222ee-ce9c-4d2a-8113-c1ca4c5c9200 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:11.560624 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a3222ee-ce9c-4d2a-8113-c1ca4c5c9200 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:11.617290 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cb83802-5936-4a1d-9950-1e1c234ce2a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:11.617892 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb83802-5936-4a1d-9950-1e1c234ce2a3 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:11.618171 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb83802-5936-4a1d-9950-1e1c234ce2a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:11.618466 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb83802-5936-4a1d-9950-1e1c234ce2a3 admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:11.619088 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cb83802-5936-4a1d-9950-1e1c234ce2a3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:11.621048 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cb83802-5936-4a1d-9950-1e1c234ce2a3 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:11.642546 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1271/2566] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:47:11 2022] POST /identity/v3/users => generated 297 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 04:47:11.646609 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c49ba992-c83e-4308-aa5e-4187b35fbee0 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:11.657726 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c49ba992-c83e-4308-aa5e-4187b35fbee0 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:11.662409 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bf27bb4-7f1e-4b30-8596-8dd3b4e21e24 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:11.663009 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bf27bb4-7f1e-4b30-8596-8dd3b4e21e24 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:11.663290 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bf27bb4-7f1e-4b30-8596-8dd3b4e21e24 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:11.663561 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bf27bb4-7f1e-4b30-8596-8dd3b4e21e24 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:11.664197 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bf27bb4-7f1e-4b30-8596-8dd3b4e21e24 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:11.665812 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bf27bb4-7f1e-4b30-8596-8dd3b4e21e24 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:11.667020 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:47:11.678691 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1296/2567] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:47:11 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:47:11.683058 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cb83802-5936-4a1d-9950-1e1c234ce2a3 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:11.694340 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cb83802-5936-4a1d-9950-1e1c234ce2a3 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:11.699378 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2486d625-9e0a-4060-b73c-0ce8bcdda2db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:11.700061 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2486d625-9e0a-4060-b73c-0ce8bcdda2db admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:11.700341 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2486d625-9e0a-4060-b73c-0ce8bcdda2db admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:11.700656 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2486d625-9e0a-4060-b73c-0ce8bcdda2db admin admin] PATH_INFO: `/v3/projects/a94cae3a97eb40b5a231502c3f538115/users/5c94dbd57bee4d5f938c8c25a1c7060a/roles/c51fe7c5ad804040879acfb54269455f` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:11.716002 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2486d625-9e0a-4060-b73c-0ce8bcdda2db admin admin] RBAC: Authorizing `identity:create_grant(project_id=a94cae3a97eb40b5a231502c3f538115, user_id=5c94dbd57bee4d5f938c8c25a1c7060a, role_id=c51fe7c5ad804040879acfb54269455f)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:11.718728 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2486d625-9e0a-4060-b73c-0ce8bcdda2db admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:11.734102 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1272/2568] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:47:11 2022] PUT /identity/v3/projects/a94cae3a97eb40b5a231502c3f538115/users/5c94dbd57bee4d5f938c8c25a1c7060a/roles/c51fe7c5ad804040879acfb54269455f => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:47:11.747097 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bf27bb4-7f1e-4b30-8596-8dd3b4e21e24 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:11.748759 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bf27bb4-7f1e-4b30-8596-8dd3b4e21e24 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:11.814453 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4662b57e-aecb-4b3c-a3d7-67d57f4370b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:11.815105 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4662b57e-aecb-4b3c-a3d7-67d57f4370b3 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:11.815352 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4662b57e-aecb-4b3c-a3d7-67d57f4370b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:11.815627 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4662b57e-aecb-4b3c-a3d7-67d57f4370b3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:11.816269 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4662b57e-aecb-4b3c-a3d7-67d57f4370b3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:11.817920 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4662b57e-aecb-4b3c-a3d7-67d57f4370b3 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:11.832545 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1297/2569] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:47:11 2022] GET /identity/v3/roles => generated 1420 bytes in 91 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:47:11.833591 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2486d625-9e0a-4060-b73c-0ce8bcdda2db admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:11.845370 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2486d625-9e0a-4060-b73c-0ce8bcdda2db admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:11.852472 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94170c0d-c2b8-4d62-922c-15a4d3beca1b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:11.853263 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94170c0d-c2b8-4d62-922c-15a4d3beca1b admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:11.853567 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94170c0d-c2b8-4d62-922c-15a4d3beca1b admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:11.853840 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94170c0d-c2b8-4d62-922c-15a4d3beca1b admin admin] PATH_INFO: `/v3/projects/a94cae3a97eb40b5a231502c3f538115/users/5c94dbd57bee4d5f938c8c25a1c7060a/roles/c51fe7c5ad804040879acfb54269455f` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:11.855771 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94170c0d-c2b8-4d62-922c-15a4d3beca1b admin admin] RBAC: Authorizing `identity:create_grant(project_id=a94cae3a97eb40b5a231502c3f538115, user_id=5c94dbd57bee4d5f938c8c25a1c7060a, role_id=c51fe7c5ad804040879acfb54269455f)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:11.858513 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94170c0d-c2b8-4d62-922c-15a4d3beca1b admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:11.869821 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1273/2570] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:47:11 2022] PUT /identity/v3/projects/a94cae3a97eb40b5a231502c3f538115/users/5c94dbd57bee4d5f938c8c25a1c7060a/roles/c51fe7c5ad804040879acfb54269455f => generated 0 bytes in 37 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:47:11.881660 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f71a27b-4331-413f-93e8-7643e441cdbf None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:11.881948 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f71a27b-4331-413f-93e8-7643e441cdbf None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:11.882222 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f71a27b-4331-413f-93e8-7643e441cdbf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:12.039380 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-9f71a27b-4331-413f-93e8-7643e441cdbf None None] MFA Rules not processed for user `5c94dbd57bee4d5f938c8c25a1c7060a`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:47:12.040607 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-9f71a27b-4331-413f-93e8-7643e441cdbf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:47:12.112350 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1298/2571] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:47:11 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 233 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:47:12.123897 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94170c0d-c2b8-4d62-922c-15a4d3beca1b admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:12.126477 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-94170c0d-c2b8-4d62-922c-15a4d3beca1b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:47:12.186335 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94170c0d-c2b8-4d62-922c-15a4d3beca1b admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:12.190802 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12c09e59-0e77-4074-accf-45ffd278c261 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ffbc136c77b84e8c87128f7247f4ec03', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:12.191416 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12c09e59-0e77-4074-accf-45ffd278c261 service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:12.191697 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12c09e59-0e77-4074-accf-45ffd278c261 service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:12.191967 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12c09e59-0e77-4074-accf-45ffd278c261 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:12.201130 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12c09e59-0e77-4074-accf-45ffd278c261 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:12.202558 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12c09e59-0e77-4074-accf-45ffd278c261 service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:12.206448 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:47:12.213762 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1274/2572] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:47:12 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:47:12.292412 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f71a27b-4331-413f-93e8-7643e441cdbf None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:12.303302 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f71a27b-4331-413f-93e8-7643e441cdbf None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:12.306941 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d609951a-262d-4f9e-9a11-66fa30512584 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ffbc136c77b84e8c87128f7247f4ec03', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:12.307538 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d609951a-262d-4f9e-9a11-66fa30512584 service cinder] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:12.307812 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d609951a-262d-4f9e-9a11-66fa30512584 service cinder] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:12.308078 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d609951a-262d-4f9e-9a11-66fa30512584 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:12.364589 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d609951a-262d-4f9e-9a11-66fa30512584 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:12.366044 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d609951a-262d-4f9e-9a11-66fa30512584 service cinder] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:12.374685 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1299/2573] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:47:12 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:47:16.297243 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12c09e59-0e77-4074-accf-45ffd278c261 service cinder] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:16.305854 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12c09e59-0e77-4074-accf-45ffd278c261 service cinder] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:16.309588 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58dc74ae-405a-477e-8900-0ec38ae8456c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ffbc136c77b84e8c87128f7247f4ec03', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:16.310173 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58dc74ae-405a-477e-8900-0ec38ae8456c service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:16.310452 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58dc74ae-405a-477e-8900-0ec38ae8456c service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:16.310720 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58dc74ae-405a-477e-8900-0ec38ae8456c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:16.312431 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-58dc74ae-405a-477e-8900-0ec38ae8456c service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:47:16.393574 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58dc74ae-405a-477e-8900-0ec38ae8456c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:16.395000 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58dc74ae-405a-477e-8900-0ec38ae8456c service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:16.413189 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1275/2574] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:47:16 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 117 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 25 04:47:22.370884 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=93 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:47:23.277012 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d609951a-262d-4f9e-9a11-66fa30512584 service cinder] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:23.338015 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d609951a-262d-4f9e-9a11-66fa30512584 service cinder] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:23.342781 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49c9db67-4769-49bd-909f-92df7ca33f66 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:23.343404 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49c9db67-4769-49bd-909f-92df7ca33f66 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:23.343691 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49c9db67-4769-49bd-909f-92df7ca33f66 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:23.343961 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49c9db67-4769-49bd-909f-92df7ca33f66 admin admin] PATH_INFO: `/v3/users/1c406f7d87504c7d9ee67302b54d5409` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:23.345307 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49c9db67-4769-49bd-909f-92df7ca33f66 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1c406f7d87504c7d9ee67302b54d5409)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:23.347181 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49c9db67-4769-49bd-909f-92df7ca33f66 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:23.392383 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-49c9db67-4769-49bd-909f-92df7ca33f66 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1c406f7d87504c7d9ee67302b54d5409'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:23.399681 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-49c9db67-4769-49bd-909f-92df7ca33f66 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1c406f7d87504c7d9ee67302b54d5409'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:23.411490 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-49c9db67-4769-49bd-909f-92df7ca33f66 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1c406f7d87504c7d9ee67302b54d5409'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:23.412127 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-49c9db67-4769-49bd-909f-92df7ca33f66 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1c406f7d87504c7d9ee67302b54d5409'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:23.418903 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-49c9db67-4769-49bd-909f-92df7ca33f66 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1c406f7d87504c7d9ee67302b54d5409'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:23.425222 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1300/2575] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:47:23 2022] DELETE /identity/v3/users/1c406f7d87504c7d9ee67302b54d5409 => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:47:23.429688 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94071d6c-f634-41ba-bbec-289f8fc60746 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:23.430176 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94071d6c-f634-41ba-bbec-289f8fc60746 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:23.430291 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94071d6c-f634-41ba-bbec-289f8fc60746 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:23.449254 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-94071d6c-f634-41ba-bbec-289f8fc60746 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:47:23.450374 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-94071d6c-f634-41ba-bbec-289f8fc60746 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:47:23.509095 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1276/2576] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:47:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:47:23.520046 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49c9db67-4769-49bd-909f-92df7ca33f66 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:23.521480 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-49c9db67-4769-49bd-909f-92df7ca33f66 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:47:23.582039 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49c9db67-4769-49bd-909f-92df7ca33f66 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:23.585782 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b251526f-bef3-4fbd-b937-02c686f86f40 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:23.586391 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b251526f-bef3-4fbd-b937-02c686f86f40 service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:23.586670 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b251526f-bef3-4fbd-b937-02c686f86f40 service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:23.586939 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b251526f-bef3-4fbd-b937-02c686f86f40 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:23.596715 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b251526f-bef3-4fbd-b937-02c686f86f40 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:23.598148 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b251526f-bef3-4fbd-b937-02c686f86f40 service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:23.607748 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1301/2577] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:47:23 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:47:23.934463 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94071d6c-f634-41ba-bbec-289f8fc60746 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:23.936031 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-94071d6c-f634-41ba-bbec-289f8fc60746 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:47:23.949075 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94071d6c-f634-41ba-bbec-289f8fc60746 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:23.954439 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab8d7bf5-61a4-4f06-aa8d-ae20b349e55d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:23.955166 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab8d7bf5-61a4-4f06-aa8d-ae20b349e55d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:23.955511 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab8d7bf5-61a4-4f06-aa8d-ae20b349e55d admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:23.955848 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab8d7bf5-61a4-4f06-aa8d-ae20b349e55d admin admin] PATH_INFO: `/v3/projects/340dd05680fc47f08828117ef16edf4f` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:23.957107 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab8d7bf5-61a4-4f06-aa8d-ae20b349e55d admin admin] RBAC: Authorizing `identity:delete_project(project_id=340dd05680fc47f08828117ef16edf4f)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:23.959249 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab8d7bf5-61a4-4f06-aa8d-ae20b349e55d admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:24.008770 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-ab8d7bf5-61a4-4f06-aa8d-ae20b349e55d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '340dd05680fc47f08828117ef16edf4f', 'request_id': 'req-ab8d7bf5-61a4-4f06-aa8d-ae20b349e55d'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:24.017448 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-ab8d7bf5-61a4-4f06-aa8d-ae20b349e55d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '340dd05680fc47f08828117ef16edf4f', 'request_id': 'req-ab8d7bf5-61a4-4f06-aa8d-ae20b349e55d'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:24.022528 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:47:24.023993 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-ab8d7bf5-61a4-4f06-aa8d-ae20b349e55d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '340dd05680fc47f08828117ef16edf4f', 'request_id': 'req-ab8d7bf5-61a4-4f06-aa8d-ae20b349e55d'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:24.038392 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-ab8d7bf5-61a4-4f06-aa8d-ae20b349e55d admin admin] The token cache is being invalidate because project 340dd05680fc47f08828117ef16edf4f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:47:24.038686 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-ab8d7bf5-61a4-4f06-aa8d-ae20b349e55d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:24.039347 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-ab8d7bf5-61a4-4f06-aa8d-ae20b349e55d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:24.040874 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1277/2578] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:47:23 2022] DELETE /identity/v3/projects/340dd05680fc47f08828117ef16edf4f => generated 0 bytes in 107 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 04:47:24.044804 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b251526f-bef3-4fbd-b937-02c686f86f40 service neutron] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:24.054699 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b251526f-bef3-4fbd-b937-02c686f86f40 service neutron] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:24.073549 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:47:24.116895 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-533e6929-31cc-46d3-b09a-c168cbcf3804 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:24.117539 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-533e6929-31cc-46d3-b09a-c168cbcf3804 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:24.117811 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-533e6929-31cc-46d3-b09a-c168cbcf3804 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:24.118087 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-533e6929-31cc-46d3-b09a-c168cbcf3804 admin admin] PATH_INFO: `/v3/users/5c94dbd57bee4d5f938c8c25a1c7060a` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:24.119218 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-533e6929-31cc-46d3-b09a-c168cbcf3804 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5c94dbd57bee4d5f938c8c25a1c7060a)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:24.121156 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-533e6929-31cc-46d3-b09a-c168cbcf3804 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:24.167812 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-533e6929-31cc-46d3-b09a-c168cbcf3804 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5c94dbd57bee4d5f938c8c25a1c7060a'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:24.175638 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-533e6929-31cc-46d3-b09a-c168cbcf3804 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5c94dbd57bee4d5f938c8c25a1c7060a'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:24.188117 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-533e6929-31cc-46d3-b09a-c168cbcf3804 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5c94dbd57bee4d5f938c8c25a1c7060a'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:24.188825 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-533e6929-31cc-46d3-b09a-c168cbcf3804 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5c94dbd57bee4d5f938c8c25a1c7060a'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:24.196177 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-533e6929-31cc-46d3-b09a-c168cbcf3804 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5c94dbd57bee4d5f938c8c25a1c7060a'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:24.202079 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1302/2579] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:47:24 2022] DELETE /identity/v3/users/5c94dbd57bee4d5f938c8c25a1c7060a => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:47:24.479432 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab8d7bf5-61a4-4f06-aa8d-ae20b349e55d admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:24.481070 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-ab8d7bf5-61a4-4f06-aa8d-ae20b349e55d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:47:24.498737 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab8d7bf5-61a4-4f06-aa8d-ae20b349e55d admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:24.559125 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b72509e-f474-48df-9b37-40f7c1983c7e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:24.559770 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b72509e-f474-48df-9b37-40f7c1983c7e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:24.560058 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b72509e-f474-48df-9b37-40f7c1983c7e admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:24.560337 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b72509e-f474-48df-9b37-40f7c1983c7e admin admin] PATH_INFO: `/v3/projects/a94cae3a97eb40b5a231502c3f538115` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:24.561504 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b72509e-f474-48df-9b37-40f7c1983c7e admin admin] RBAC: Authorizing `identity:delete_project(project_id=a94cae3a97eb40b5a231502c3f538115)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:24.563483 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b72509e-f474-48df-9b37-40f7c1983c7e admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:24.613184 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-3b72509e-f474-48df-9b37-40f7c1983c7e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a94cae3a97eb40b5a231502c3f538115', 'request_id': 'req-3b72509e-f474-48df-9b37-40f7c1983c7e'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:24.621108 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-3b72509e-f474-48df-9b37-40f7c1983c7e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a94cae3a97eb40b5a231502c3f538115', 'request_id': 'req-3b72509e-f474-48df-9b37-40f7c1983c7e'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:24.627379 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-3b72509e-f474-48df-9b37-40f7c1983c7e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a94cae3a97eb40b5a231502c3f538115', 'request_id': 'req-3b72509e-f474-48df-9b37-40f7c1983c7e'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:24.638697 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-3b72509e-f474-48df-9b37-40f7c1983c7e admin admin] The token cache is being invalidate because project a94cae3a97eb40b5a231502c3f538115 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:47:24.638988 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-3b72509e-f474-48df-9b37-40f7c1983c7e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:24.639640 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-3b72509e-f474-48df-9b37-40f7c1983c7e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:24.641574 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1278/2580] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:47:24 2022] DELETE /identity/v3/projects/a94cae3a97eb40b5a231502c3f538115 => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:47:24.646324 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d088c3-de2c-43fb-91d2-aafbdce36e4d None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:24.646834 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d088c3-de2c-43fb-91d2-aafbdce36e4d None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:24.646950 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d088c3-de2c-43fb-91d2-aafbdce36e4d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:24.667426 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-a3d088c3-de2c-43fb-91d2-aafbdce36e4d None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:47:24.668621 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-a3d088c3-de2c-43fb-91d2-aafbdce36e4d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:47:24.729760 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1303/2581] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:47:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:47:24.738643 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c388c8e4-2729-4a51-a7a1-a0e9a8464f69 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:24.738986 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c388c8e4-2729-4a51-a7a1-a0e9a8464f69 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:24.739333 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c388c8e4-2729-4a51-a7a1-a0e9a8464f69 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:24.761739 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-c388c8e4-2729-4a51-a7a1-a0e9a8464f69 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:47:24.762903 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-c388c8e4-2729-4a51-a7a1-a0e9a8464f69 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:47:24.773042 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1279/2582] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:47:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:47:24.778149 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3d088c3-de2c-43fb-91d2-aafbdce36e4d None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:24.790495 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3d088c3-de2c-43fb-91d2-aafbdce36e4d None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:24.794966 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e3f47ed-6bdf-4a15-83db-691da8150ee1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:24.795578 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e3f47ed-6bdf-4a15-83db-691da8150ee1 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:24.795855 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e3f47ed-6bdf-4a15-83db-691da8150ee1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:24.796125 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e3f47ed-6bdf-4a15-83db-691da8150ee1 admin admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:24.796855 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e3f47ed-6bdf-4a15-83db-691da8150ee1 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:47:24.797400 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e3f47ed-6bdf-4a15-83db-691da8150ee1 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:24.799099 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e3f47ed-6bdf-4a15-83db-691da8150ee1 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:24.813979 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1304/2583] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Sep 25 04:47:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:47:24.816149 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c388c8e4-2729-4a51-a7a1-a0e9a8464f69 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:24.826590 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c388c8e4-2729-4a51-a7a1-a0e9a8464f69 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:24.832039 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3f2b956-dab6-4272-a696-6e88561e0119 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:24.832665 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3f2b956-dab6-4272-a696-6e88561e0119 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:24.832954 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3f2b956-dab6-4272-a696-6e88561e0119 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:24.833225 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3f2b956-dab6-4272-a696-6e88561e0119 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:24.833852 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3f2b956-dab6-4272-a696-6e88561e0119 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:24.835771 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3f2b956-dab6-4272-a696-6e88561e0119 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:24.864222 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1280/2584] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:47:24 2022] POST /identity/v3/projects => generated 363 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:47:24.868683 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e3f47ed-6bdf-4a15-83db-691da8150ee1 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:24.880092 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e3f47ed-6bdf-4a15-83db-691da8150ee1 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:24.910347 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:47:24.952743 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-168fff2d-206e-45f4-911b-74be73994876 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:24.954212 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-168fff2d-206e-45f4-911b-74be73994876 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:24.954720 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-168fff2d-206e-45f4-911b-74be73994876 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:24.955042 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-168fff2d-206e-45f4-911b-74be73994876 admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:24.955738 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-168fff2d-206e-45f4-911b-74be73994876 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:24.957921 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-168fff2d-206e-45f4-911b-74be73994876 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:24.983612 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1305/2585] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:47:24 2022] POST /identity/v3/users => generated 282 bytes in 116 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:47:24.987053 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3f2b956-dab6-4272-a696-6e88561e0119 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:24.998607 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3f2b956-dab6-4272-a696-6e88561e0119 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:25.003860 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47c49484-704a-4634-861d-a8014c6293f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:25.004582 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c49484-704a-4634-861d-a8014c6293f3 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:25.005006 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c49484-704a-4634-861d-a8014c6293f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:25.005406 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c49484-704a-4634-861d-a8014c6293f3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:25.006142 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47c49484-704a-4634-861d-a8014c6293f3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:25.007754 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47c49484-704a-4634-861d-a8014c6293f3 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:25.029182 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1281/2586] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:47:24 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:47:25.030519 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-168fff2d-206e-45f4-911b-74be73994876 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:25.052767 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-168fff2d-206e-45f4-911b-74be73994876 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:25.062849 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eec1b4ba-3257-4827-bc36-c5b4323a2cd3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:25.063784 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eec1b4ba-3257-4827-bc36-c5b4323a2cd3 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:25.064250 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eec1b4ba-3257-4827-bc36-c5b4323a2cd3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:25.064663 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eec1b4ba-3257-4827-bc36-c5b4323a2cd3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:25.065412 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eec1b4ba-3257-4827-bc36-c5b4323a2cd3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:25.067100 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eec1b4ba-3257-4827-bc36-c5b4323a2cd3 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:25.088929 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1306/2587] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:47:25 2022] GET /identity/v3/roles => generated 1420 bytes in 59 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:47:25.090761 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47c49484-704a-4634-861d-a8014c6293f3 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:25.106725 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47c49484-704a-4634-861d-a8014c6293f3 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:25.131891 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b52a7e63-0ff8-4ff6-bf2d-04ca1085efb2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:25.132574 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b52a7e63-0ff8-4ff6-bf2d-04ca1085efb2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:25.132890 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b52a7e63-0ff8-4ff6-bf2d-04ca1085efb2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:25.133162 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b52a7e63-0ff8-4ff6-bf2d-04ca1085efb2 admin admin] PATH_INFO: `/v3/projects/a83ae87438b94170ae9e24cf3fc17902/users/b6f8383d3b494ce9a73fbf262af9f981/roles/8f6cc574056c4278bc9778f14dc4b411` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:25.153394 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b52a7e63-0ff8-4ff6-bf2d-04ca1085efb2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a83ae87438b94170ae9e24cf3fc17902, user_id=b6f8383d3b494ce9a73fbf262af9f981, role_id=8f6cc574056c4278bc9778f14dc4b411)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:25.158134 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b52a7e63-0ff8-4ff6-bf2d-04ca1085efb2 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:25.186825 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1282/2588] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:47:25 2022] PUT /identity/v3/projects/a83ae87438b94170ae9e24cf3fc17902/users/b6f8383d3b494ce9a73fbf262af9f981/roles/8f6cc574056c4278bc9778f14dc4b411 => generated 0 bytes in 86 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:47:25.187041 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a40ab7f-f8a7-42f2-ab24-f39d73398e24 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:25.187833 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a40ab7f-f8a7-42f2-ab24-f39d73398e24 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:25.188278 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a40ab7f-f8a7-42f2-ab24-f39d73398e24 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:25.227949 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-8a40ab7f-f8a7-42f2-ab24-f39d73398e24 None None] MFA Rules not processed for user `b6f8383d3b494ce9a73fbf262af9f981`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:47:25.229673 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-8a40ab7f-f8a7-42f2-ab24-f39d73398e24 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:47:25.333360 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1307/2589] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:47:25 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 04:47:25.334870 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b52a7e63-0ff8-4ff6-bf2d-04ca1085efb2 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:25.361211 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:47:25.415235 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b52a7e63-0ff8-4ff6-bf2d-04ca1085efb2 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:25.420440 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10401a5d-975b-449b-b8f7-9614297a64c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:25.421291 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10401a5d-975b-449b-b8f7-9614297a64c4 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:25.422302 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10401a5d-975b-449b-b8f7-9614297a64c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:25.422747 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10401a5d-975b-449b-b8f7-9614297a64c4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:25.423486 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10401a5d-975b-449b-b8f7-9614297a64c4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:25.425537 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10401a5d-975b-449b-b8f7-9614297a64c4 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:25.459043 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1283/2590] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:47:25 2022] POST /identity/v3/projects => generated 363 bytes in 122 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:47:25.460719 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a40ab7f-f8a7-42f2-ab24-f39d73398e24 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:25.479678 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a40ab7f-f8a7-42f2-ab24-f39d73398e24 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:25.547636 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6babbbe1-7935-41c9-b51a-4c424401ae28 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:25.548394 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6babbbe1-7935-41c9-b51a-4c424401ae28 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:25.548818 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6babbbe1-7935-41c9-b51a-4c424401ae28 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:25.549196 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6babbbe1-7935-41c9-b51a-4c424401ae28 admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:25.550251 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6babbbe1-7935-41c9-b51a-4c424401ae28 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:25.551968 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6babbbe1-7935-41c9-b51a-4c424401ae28 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:25.580193 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1308/2591] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:47:25 2022] POST /identity/v3/users => generated 288 bytes in 117 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:47:25.581820 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10401a5d-975b-449b-b8f7-9614297a64c4 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:25.604660 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10401a5d-975b-449b-b8f7-9614297a64c4 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:25.615653 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51e443d1-7d23-4785-82c7-1e6ca86e17d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:25.615852 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51e443d1-7d23-4785-82c7-1e6ca86e17d3 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:25.616149 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51e443d1-7d23-4785-82c7-1e6ca86e17d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:25.616429 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51e443d1-7d23-4785-82c7-1e6ca86e17d3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:25.617801 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51e443d1-7d23-4785-82c7-1e6ca86e17d3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:25.619097 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51e443d1-7d23-4785-82c7-1e6ca86e17d3 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:25.632669 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1284/2592] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:47:25 2022] GET /identity/v3/roles => generated 1420 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:47:25.635615 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6babbbe1-7935-41c9-b51a-4c424401ae28 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:25.647098 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6babbbe1-7935-41c9-b51a-4c424401ae28 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:25.651915 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed154801-cec9-41b1-babf-98e753a19d5d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:25.652628 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed154801-cec9-41b1-babf-98e753a19d5d admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:25.652951 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed154801-cec9-41b1-babf-98e753a19d5d admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:25.653228 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed154801-cec9-41b1-babf-98e753a19d5d admin admin] PATH_INFO: `/v3/projects/80c3928408864171a5b2e04dcbe1d335/users/a1cfec1a25b748e0a84a78ed695f2174/roles/c51fe7c5ad804040879acfb54269455f` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:25.673726 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed154801-cec9-41b1-babf-98e753a19d5d admin admin] RBAC: Authorizing `identity:create_grant(project_id=80c3928408864171a5b2e04dcbe1d335, user_id=a1cfec1a25b748e0a84a78ed695f2174, role_id=c51fe7c5ad804040879acfb54269455f)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:25.678326 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed154801-cec9-41b1-babf-98e753a19d5d admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:25.696990 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1309/2593] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:47:25 2022] PUT /identity/v3/projects/80c3928408864171a5b2e04dcbe1d335/users/a1cfec1a25b748e0a84a78ed695f2174/roles/c51fe7c5ad804040879acfb54269455f => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 04:47:25.698909 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51e443d1-7d23-4785-82c7-1e6ca86e17d3 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:25.721225 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51e443d1-7d23-4785-82c7-1e6ca86e17d3 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:25.796271 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f25c10ca-0841-476a-91d8-fd71ee3943a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:25.796732 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f25c10ca-0841-476a-91d8-fd71ee3943a4 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:25.797067 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f25c10ca-0841-476a-91d8-fd71ee3943a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:25.797360 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f25c10ca-0841-476a-91d8-fd71ee3943a4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:25.798395 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f25c10ca-0841-476a-91d8-fd71ee3943a4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:25.803057 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f25c10ca-0841-476a-91d8-fd71ee3943a4 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:25.814545 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1285/2594] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:47:25 2022] GET /identity/v3/roles => generated 1420 bytes in 116 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:47:25.818656 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed154801-cec9-41b1-babf-98e753a19d5d admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:25.824172 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:47:25.834324 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed154801-cec9-41b1-babf-98e753a19d5d admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:25.840780 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7db3a3fe-bf03-4820-9229-a8624a015f13 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:25.841736 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7db3a3fe-bf03-4820-9229-a8624a015f13 admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:25.842137 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7db3a3fe-bf03-4820-9229-a8624a015f13 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:25.842485 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7db3a3fe-bf03-4820-9229-a8624a015f13 admin admin] PATH_INFO: `/v3/projects/80c3928408864171a5b2e04dcbe1d335/users/a1cfec1a25b748e0a84a78ed695f2174/roles/c51fe7c5ad804040879acfb54269455f` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:25.844866 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7db3a3fe-bf03-4820-9229-a8624a015f13 admin admin] RBAC: Authorizing `identity:create_grant(project_id=80c3928408864171a5b2e04dcbe1d335, user_id=a1cfec1a25b748e0a84a78ed695f2174, role_id=c51fe7c5ad804040879acfb54269455f)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:25.847522 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7db3a3fe-bf03-4820-9229-a8624a015f13 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:25.859051 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1310/2595] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:47:25 2022] PUT /identity/v3/projects/80c3928408864171a5b2e04dcbe1d335/users/a1cfec1a25b748e0a84a78ed695f2174/roles/c51fe7c5ad804040879acfb54269455f => generated 0 bytes in 41 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:47:25.881580 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70f288ea-25a3-4acc-ae7b-ac995b74de83 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:25.881885 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70f288ea-25a3-4acc-ae7b-ac995b74de83 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:25.882151 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70f288ea-25a3-4acc-ae7b-ac995b74de83 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:25.912842 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-70f288ea-25a3-4acc-ae7b-ac995b74de83 None None] MFA Rules not processed for user `a1cfec1a25b748e0a84a78ed695f2174`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:47:25.914065 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-70f288ea-25a3-4acc-ae7b-ac995b74de83 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:47:25.993470 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1286/2596] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:47:25 2022] POST /identity/v3/auth/tokens => generated 3307 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:47:26.006609 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7db3a3fe-bf03-4820-9229-a8624a015f13 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:26.008319 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-7db3a3fe-bf03-4820-9229-a8624a015f13 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:47:26.077674 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7db3a3fe-bf03-4820-9229-a8624a015f13 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:26.082094 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c67dc86c-f94c-45f6-8642-0f6d27126a81 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ffbc136c77b84e8c87128f7247f4ec03', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:26.082887 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c67dc86c-f94c-45f6-8642-0f6d27126a81 service cinder] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:26.083164 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c67dc86c-f94c-45f6-8642-0f6d27126a81 service cinder] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:26.083432 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c67dc86c-f94c-45f6-8642-0f6d27126a81 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:26.095744 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c67dc86c-f94c-45f6-8642-0f6d27126a81 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:26.096740 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c67dc86c-f94c-45f6-8642-0f6d27126a81 service cinder] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:26.108757 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1311/2597] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:47:26 2022] GET /identity/v3/auth/tokens => generated 3307 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:47:26.229484 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70f288ea-25a3-4acc-ae7b-ac995b74de83 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:26.247031 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70f288ea-25a3-4acc-ae7b-ac995b74de83 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:26.267060 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b124fa1-ae04-4356-9b07-46ae07111f60 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ffbc136c77b84e8c87128f7247f4ec03', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:26.267975 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b124fa1-ae04-4356-9b07-46ae07111f60 service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:26.268332 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b124fa1-ae04-4356-9b07-46ae07111f60 service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:26.268706 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b124fa1-ae04-4356-9b07-46ae07111f60 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:26.285896 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:47:26.337230 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b124fa1-ae04-4356-9b07-46ae07111f60 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:26.338653 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b124fa1-ae04-4356-9b07-46ae07111f60 service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:26.348465 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1287/2598] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:47:26 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 120 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:47:29.172078 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c67dc86c-f94c-45f6-8642-0f6d27126a81 service cinder] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:29.236040 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c67dc86c-f94c-45f6-8642-0f6d27126a81 service cinder] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:29.241045 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5d0d91c-0bc3-4f1c-8d8c-33ec11f1210b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:29.241696 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5d0d91c-0bc3-4f1c-8d8c-33ec11f1210b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:29.241969 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5d0d91c-0bc3-4f1c-8d8c-33ec11f1210b admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:29.242235 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5d0d91c-0bc3-4f1c-8d8c-33ec11f1210b admin admin] PATH_INFO: `/v3/users/b6f8383d3b494ce9a73fbf262af9f981` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:29.243349 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5d0d91c-0bc3-4f1c-8d8c-33ec11f1210b admin admin] RBAC: Authorizing `identity:delete_user(user_id=b6f8383d3b494ce9a73fbf262af9f981)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:29.245286 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5d0d91c-0bc3-4f1c-8d8c-33ec11f1210b admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:29.291102 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-d5d0d91c-0bc3-4f1c-8d8c-33ec11f1210b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b6f8383d3b494ce9a73fbf262af9f981'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:29.299383 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-d5d0d91c-0bc3-4f1c-8d8c-33ec11f1210b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b6f8383d3b494ce9a73fbf262af9f981'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:29.313710 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-d5d0d91c-0bc3-4f1c-8d8c-33ec11f1210b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b6f8383d3b494ce9a73fbf262af9f981'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:29.314403 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-d5d0d91c-0bc3-4f1c-8d8c-33ec11f1210b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b6f8383d3b494ce9a73fbf262af9f981'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:29.321399 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-d5d0d91c-0bc3-4f1c-8d8c-33ec11f1210b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b6f8383d3b494ce9a73fbf262af9f981'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:29.327423 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1312/2599] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:47:29 2022] DELETE /identity/v3/users/b6f8383d3b494ce9a73fbf262af9f981 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:47:29.332006 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7af20f06-a71c-47f2-83c6-637bd63021f5 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:29.332501 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7af20f06-a71c-47f2-83c6-637bd63021f5 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:29.332604 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7af20f06-a71c-47f2-83c6-637bd63021f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:29.351723 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-7af20f06-a71c-47f2-83c6-637bd63021f5 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:47:29.352978 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-7af20f06-a71c-47f2-83c6-637bd63021f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:47:29.421023 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1288/2600] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:47:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:47:29.430942 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5d0d91c-0bc3-4f1c-8d8c-33ec11f1210b admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:29.433448 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-d5d0d91c-0bc3-4f1c-8d8c-33ec11f1210b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:47:29.449765 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:47:29.509949 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5d0d91c-0bc3-4f1c-8d8c-33ec11f1210b admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:29.513954 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50e852d4-8bca-4a2f-a0e7-85554f892458 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:29.514574 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e852d4-8bca-4a2f-a0e7-85554f892458 service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:29.514863 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e852d4-8bca-4a2f-a0e7-85554f892458 service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:29.515202 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e852d4-8bca-4a2f-a0e7-85554f892458 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:29.525287 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50e852d4-8bca-4a2f-a0e7-85554f892458 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:29.526825 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50e852d4-8bca-4a2f-a0e7-85554f892458 service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:29.537230 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1313/2601] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:47:29 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:47:29.808718 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7af20f06-a71c-47f2-83c6-637bd63021f5 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:29.810152 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-7af20f06-a71c-47f2-83c6-637bd63021f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:47:29.823100 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7af20f06-a71c-47f2-83c6-637bd63021f5 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:29.828501 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e69b66c-4047-4cf2-97d2-b50185b8a5a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:29.829187 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e69b66c-4047-4cf2-97d2-b50185b8a5a0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:29.829471 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e69b66c-4047-4cf2-97d2-b50185b8a5a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:29.830127 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e69b66c-4047-4cf2-97d2-b50185b8a5a0 admin admin] PATH_INFO: `/v3/projects/a83ae87438b94170ae9e24cf3fc17902` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:29.830967 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e69b66c-4047-4cf2-97d2-b50185b8a5a0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a83ae87438b94170ae9e24cf3fc17902)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:29.832931 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e69b66c-4047-4cf2-97d2-b50185b8a5a0 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:29.880151 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-8e69b66c-4047-4cf2-97d2-b50185b8a5a0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a83ae87438b94170ae9e24cf3fc17902', 'request_id': 'req-8e69b66c-4047-4cf2-97d2-b50185b8a5a0'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:29.888413 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-8e69b66c-4047-4cf2-97d2-b50185b8a5a0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a83ae87438b94170ae9e24cf3fc17902', 'request_id': 'req-8e69b66c-4047-4cf2-97d2-b50185b8a5a0'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:29.894138 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-8e69b66c-4047-4cf2-97d2-b50185b8a5a0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a83ae87438b94170ae9e24cf3fc17902', 'request_id': 'req-8e69b66c-4047-4cf2-97d2-b50185b8a5a0'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:29.905248 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-8e69b66c-4047-4cf2-97d2-b50185b8a5a0 admin admin] The token cache is being invalidate because project a83ae87438b94170ae9e24cf3fc17902 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:47:29.905605 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-8e69b66c-4047-4cf2-97d2-b50185b8a5a0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:29.907012 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-8e69b66c-4047-4cf2-97d2-b50185b8a5a0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:29.908544 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1289/2602] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:47:29 2022] DELETE /identity/v3/projects/a83ae87438b94170ae9e24cf3fc17902 => generated 0 bytes in 100 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 04:47:29.912734 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50e852d4-8bca-4a2f-a0e7-85554f892458 service neutron] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:29.922993 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50e852d4-8bca-4a2f-a0e7-85554f892458 service neutron] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:29.978514 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-181732ca-5761-4b9c-a0a4-d8a82a8dbfcf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:29.979131 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-181732ca-5761-4b9c-a0a4-d8a82a8dbfcf admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:29.979409 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-181732ca-5761-4b9c-a0a4-d8a82a8dbfcf admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:29.979679 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-181732ca-5761-4b9c-a0a4-d8a82a8dbfcf admin admin] PATH_INFO: `/v3/users/a1cfec1a25b748e0a84a78ed695f2174` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:29.980827 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-181732ca-5761-4b9c-a0a4-d8a82a8dbfcf admin admin] RBAC: Authorizing `identity:delete_user(user_id=a1cfec1a25b748e0a84a78ed695f2174)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:29.982723 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-181732ca-5761-4b9c-a0a4-d8a82a8dbfcf admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:30.028354 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-181732ca-5761-4b9c-a0a4-d8a82a8dbfcf admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a1cfec1a25b748e0a84a78ed695f2174'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:30.035945 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-181732ca-5761-4b9c-a0a4-d8a82a8dbfcf admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a1cfec1a25b748e0a84a78ed695f2174'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:30.047706 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-181732ca-5761-4b9c-a0a4-d8a82a8dbfcf admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a1cfec1a25b748e0a84a78ed695f2174'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:30.048337 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-181732ca-5761-4b9c-a0a4-d8a82a8dbfcf admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a1cfec1a25b748e0a84a78ed695f2174'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:30.055007 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-181732ca-5761-4b9c-a0a4-d8a82a8dbfcf admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a1cfec1a25b748e0a84a78ed695f2174'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:30.060731 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1314/2603] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:47:29 2022] DELETE /identity/v3/users/a1cfec1a25b748e0a84a78ed695f2174 => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:47:30.335242 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e69b66c-4047-4cf2-97d2-b50185b8a5a0 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:30.336945 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-8e69b66c-4047-4cf2-97d2-b50185b8a5a0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:47:30.376167 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e69b66c-4047-4cf2-97d2-b50185b8a5a0 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:30.417690 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:47:30.454451 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88d6b29e-9717-4cb7-a41c-47b693f74147 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:30.455084 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88d6b29e-9717-4cb7-a41c-47b693f74147 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:30.455362 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88d6b29e-9717-4cb7-a41c-47b693f74147 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:30.455632 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88d6b29e-9717-4cb7-a41c-47b693f74147 admin admin] PATH_INFO: `/v3/projects/80c3928408864171a5b2e04dcbe1d335` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:30.456767 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88d6b29e-9717-4cb7-a41c-47b693f74147 admin admin] RBAC: Authorizing `identity:delete_project(project_id=80c3928408864171a5b2e04dcbe1d335)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:30.458662 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88d6b29e-9717-4cb7-a41c-47b693f74147 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:30.504598 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-88d6b29e-9717-4cb7-a41c-47b693f74147 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '80c3928408864171a5b2e04dcbe1d335', 'request_id': 'req-88d6b29e-9717-4cb7-a41c-47b693f74147'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:30.512567 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-88d6b29e-9717-4cb7-a41c-47b693f74147 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '80c3928408864171a5b2e04dcbe1d335', 'request_id': 'req-88d6b29e-9717-4cb7-a41c-47b693f74147'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:30.519625 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-88d6b29e-9717-4cb7-a41c-47b693f74147 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '80c3928408864171a5b2e04dcbe1d335', 'request_id': 'req-88d6b29e-9717-4cb7-a41c-47b693f74147'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:30.531497 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-88d6b29e-9717-4cb7-a41c-47b693f74147 admin admin] The token cache is being invalidate because project 80c3928408864171a5b2e04dcbe1d335 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:47:30.531747 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-88d6b29e-9717-4cb7-a41c-47b693f74147 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:30.532469 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-88d6b29e-9717-4cb7-a41c-47b693f74147 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:30.537219 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1290/2604] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:47:30 2022] DELETE /identity/v3/projects/80c3928408864171a5b2e04dcbe1d335 => generated 0 bytes in 201 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:47:30.541330 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6ec96ad-6ca0-439c-908c-63f10aaef78a None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:30.541828 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6ec96ad-6ca0-439c-908c-63f10aaef78a None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:30.542050 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6ec96ad-6ca0-439c-908c-63f10aaef78a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:30.562810 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-d6ec96ad-6ca0-439c-908c-63f10aaef78a None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:47:30.564102 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-d6ec96ad-6ca0-439c-908c-63f10aaef78a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:47:30.595692 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:47:30.629634 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1315/2605] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:47:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:47:30.637485 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d504eadd-013b-4957-b002-cc1e468c13bf None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:30.637905 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d504eadd-013b-4957-b002-cc1e468c13bf None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:30.638283 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d504eadd-013b-4957-b002-cc1e468c13bf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:30.660080 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-d504eadd-013b-4957-b002-cc1e468c13bf None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:47:30.661349 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-d504eadd-013b-4957-b002-cc1e468c13bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:47:30.669346 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1291/2606] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:47:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:47:30.673598 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6ec96ad-6ca0-439c-908c-63f10aaef78a None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:30.683064 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6ec96ad-6ca0-439c-908c-63f10aaef78a None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:30.688488 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a8acb2b-437a-41c6-a01f-3204ec57fc91 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:30.689129 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a8acb2b-437a-41c6-a01f-3204ec57fc91 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:30.689410 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a8acb2b-437a-41c6-a01f-3204ec57fc91 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:30.689682 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a8acb2b-437a-41c6-a01f-3204ec57fc91 admin admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:30.690321 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a8acb2b-437a-41c6-a01f-3204ec57fc91 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:47:30.690628 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a8acb2b-437a-41c6-a01f-3204ec57fc91 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:30.692235 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a8acb2b-437a-41c6-a01f-3204ec57fc91 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:30.704497 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1316/2607] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Sep 25 04:47:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:47:30.708625 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d504eadd-013b-4957-b002-cc1e468c13bf None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:30.718176 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d504eadd-013b-4957-b002-cc1e468c13bf None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:30.723302 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e7ab22c-5906-454f-973d-304b1460576b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:30.723925 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e7ab22c-5906-454f-973d-304b1460576b admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:30.724216 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e7ab22c-5906-454f-973d-304b1460576b admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:30.724489 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e7ab22c-5906-454f-973d-304b1460576b admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:30.725161 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e7ab22c-5906-454f-973d-304b1460576b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:30.727100 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e7ab22c-5906-454f-973d-304b1460576b admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:30.758035 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1292/2608] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:47:30 2022] POST /identity/v3/projects => generated 383 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:47:30.759092 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a8acb2b-437a-41c6-a01f-3204ec57fc91 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:30.769783 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a8acb2b-437a-41c6-a01f-3204ec57fc91 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:30.826644 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2a927cb-8f9e-47af-b2a1-e5954c7dab96 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:30.827411 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a927cb-8f9e-47af-b2a1-e5954c7dab96 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:30.827718 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a927cb-8f9e-47af-b2a1-e5954c7dab96 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:30.828013 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a927cb-8f9e-47af-b2a1-e5954c7dab96 admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:30.828790 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2a927cb-8f9e-47af-b2a1-e5954c7dab96 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:30.830691 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2a927cb-8f9e-47af-b2a1-e5954c7dab96 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:30.852481 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1317/2609] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:47:30 2022] POST /identity/v3/users => generated 292 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 04:47:30.856579 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e7ab22c-5906-454f-973d-304b1460576b admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:30.867692 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e7ab22c-5906-454f-973d-304b1460576b admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:30.872456 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db3f8102-91d8-4cbb-bb4c-0509f9741ac6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:30.873081 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db3f8102-91d8-4cbb-bb4c-0509f9741ac6 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:30.873365 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db3f8102-91d8-4cbb-bb4c-0509f9741ac6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:30.873640 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db3f8102-91d8-4cbb-bb4c-0509f9741ac6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:30.874292 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db3f8102-91d8-4cbb-bb4c-0509f9741ac6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:30.876690 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db3f8102-91d8-4cbb-bb4c-0509f9741ac6 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:30.887312 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1293/2610] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:47:30 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:47:30.891681 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2a927cb-8f9e-47af-b2a1-e5954c7dab96 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:30.902116 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2a927cb-8f9e-47af-b2a1-e5954c7dab96 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:30.907494 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4194ecd-faa5-464f-baff-eac64b74233a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:30.908029 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4194ecd-faa5-464f-baff-eac64b74233a admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:30.908312 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4194ecd-faa5-464f-baff-eac64b74233a admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:30.908672 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4194ecd-faa5-464f-baff-eac64b74233a admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:30.909357 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4194ecd-faa5-464f-baff-eac64b74233a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:30.911076 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4194ecd-faa5-464f-baff-eac64b74233a admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:30.923148 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1318/2611] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:47:30 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:47:30.928051 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db3f8102-91d8-4cbb-bb4c-0509f9741ac6 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:30.939178 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db3f8102-91d8-4cbb-bb4c-0509f9741ac6 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:30.943988 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba2bda5d-612f-40d3-8f5b-f354b3d65929 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:30.945014 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba2bda5d-612f-40d3-8f5b-f354b3d65929 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:30.945014 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba2bda5d-612f-40d3-8f5b-f354b3d65929 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:30.945346 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba2bda5d-612f-40d3-8f5b-f354b3d65929 admin admin] PATH_INFO: `/v3/projects/e17a7ece84f1424cbef9a3a9924b7f93/users/d93b21e8ea94486c8527e31b84c635fa/roles/8f6cc574056c4278bc9778f14dc4b411` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:30.961668 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba2bda5d-612f-40d3-8f5b-f354b3d65929 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e17a7ece84f1424cbef9a3a9924b7f93, user_id=d93b21e8ea94486c8527e31b84c635fa, role_id=8f6cc574056c4278bc9778f14dc4b411)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:30.964653 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba2bda5d-612f-40d3-8f5b-f354b3d65929 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:30.978116 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1294/2612] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:47:30 2022] PUT /identity/v3/projects/e17a7ece84f1424cbef9a3a9924b7f93/users/d93b21e8ea94486c8527e31b84c635fa/roles/8f6cc574056c4278bc9778f14dc4b411 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 04:47:30.989316 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9217a810-7e3b-49cd-808f-50dfd186c11e None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:30.989772 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9217a810-7e3b-49cd-808f-50dfd186c11e None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:30.990954 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9217a810-7e3b-49cd-808f-50dfd186c11e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:31.023677 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-9217a810-7e3b-49cd-808f-50dfd186c11e None None] MFA Rules not processed for user `d93b21e8ea94486c8527e31b84c635fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:47:31.025077 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-9217a810-7e3b-49cd-808f-50dfd186c11e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:47:31.092822 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1319/2613] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:47:30 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:47:31.097278 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba2bda5d-612f-40d3-8f5b-f354b3d65929 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:31.160478 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba2bda5d-612f-40d3-8f5b-f354b3d65929 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:31.165153 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14a67459-9964-466c-bbba-7eee9ceff87a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:31.165761 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14a67459-9964-466c-bbba-7eee9ceff87a admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:31.166042 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14a67459-9964-466c-bbba-7eee9ceff87a admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:31.166316 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14a67459-9964-466c-bbba-7eee9ceff87a admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:31.166952 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14a67459-9964-466c-bbba-7eee9ceff87a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:31.168872 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14a67459-9964-466c-bbba-7eee9ceff87a admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:31.174783 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:47:31.198093 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1295/2614] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:47:31 2022] POST /identity/v3/projects => generated 383 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:47:31.202300 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9217a810-7e3b-49cd-808f-50dfd186c11e None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:31.213084 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9217a810-7e3b-49cd-808f-50dfd186c11e None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:31.241460 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:47:31.272912 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8d15ac4-bcc9-4967-9655-fe10aa1b5a5f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:31.273531 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8d15ac4-bcc9-4967-9655-fe10aa1b5a5f admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:31.273813 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8d15ac4-bcc9-4967-9655-fe10aa1b5a5f admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:31.274089 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8d15ac4-bcc9-4967-9655-fe10aa1b5a5f admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:31.274730 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8d15ac4-bcc9-4967-9655-fe10aa1b5a5f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:31.276687 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8d15ac4-bcc9-4967-9655-fe10aa1b5a5f admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:31.298690 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1320/2615] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:47:31 2022] POST /identity/v3/users => generated 298 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:47:31.302735 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14a67459-9964-466c-bbba-7eee9ceff87a admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:31.313301 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14a67459-9964-466c-bbba-7eee9ceff87a admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:31.318477 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af024418-63c7-4342-a9c6-b7e18e17fe5f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:31.319084 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af024418-63c7-4342-a9c6-b7e18e17fe5f admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:31.319366 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af024418-63c7-4342-a9c6-b7e18e17fe5f admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:31.319642 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af024418-63c7-4342-a9c6-b7e18e17fe5f admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:31.320287 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af024418-63c7-4342-a9c6-b7e18e17fe5f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:31.321920 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af024418-63c7-4342-a9c6-b7e18e17fe5f admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:31.332009 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1296/2616] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:47:31 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:47:31.336027 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8d15ac4-bcc9-4967-9655-fe10aa1b5a5f admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:31.347111 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8d15ac4-bcc9-4967-9655-fe10aa1b5a5f admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:31.351730 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5ef95bf-44fc-4802-9a88-e47870bf1e2f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:31.352428 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5ef95bf-44fc-4802-9a88-e47870bf1e2f admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:31.352747 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5ef95bf-44fc-4802-9a88-e47870bf1e2f admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:31.353021 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5ef95bf-44fc-4802-9a88-e47870bf1e2f admin admin] PATH_INFO: `/v3/projects/c8bfec29991445f5aadb0a33910eb92b/users/878e8c69fa8d4aadac3968a602c603c5/roles/c51fe7c5ad804040879acfb54269455f` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:31.368167 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5ef95bf-44fc-4802-9a88-e47870bf1e2f admin admin] RBAC: Authorizing `identity:create_grant(project_id=c8bfec29991445f5aadb0a33910eb92b, user_id=878e8c69fa8d4aadac3968a602c603c5, role_id=c51fe7c5ad804040879acfb54269455f)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:31.370823 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5ef95bf-44fc-4802-9a88-e47870bf1e2f admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:31.383061 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1321/2617] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:47:31 2022] PUT /identity/v3/projects/c8bfec29991445f5aadb0a33910eb92b/users/878e8c69fa8d4aadac3968a602c603c5/roles/c51fe7c5ad804040879acfb54269455f => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:47:31.387157 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af024418-63c7-4342-a9c6-b7e18e17fe5f admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:31.397218 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af024418-63c7-4342-a9c6-b7e18e17fe5f admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:31.454816 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3eb68f8-c7c5-4e8b-b25d-3ff45292842a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:31.455347 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3eb68f8-c7c5-4e8b-b25d-3ff45292842a admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:31.455722 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3eb68f8-c7c5-4e8b-b25d-3ff45292842a admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:31.456174 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3eb68f8-c7c5-4e8b-b25d-3ff45292842a admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:31.456768 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3eb68f8-c7c5-4e8b-b25d-3ff45292842a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:31.458390 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3eb68f8-c7c5-4e8b-b25d-3ff45292842a admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:31.472619 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1297/2618] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:47:31 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:47:31.473783 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5ef95bf-44fc-4802-9a88-e47870bf1e2f admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:31.485156 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5ef95bf-44fc-4802-9a88-e47870bf1e2f admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:31.489854 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12ea07f8-6ca7-4f1d-8dbf-f85039f3317c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:31.490509 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ea07f8-6ca7-4f1d-8dbf-f85039f3317c admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:31.490805 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ea07f8-6ca7-4f1d-8dbf-f85039f3317c admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:31.491050 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ea07f8-6ca7-4f1d-8dbf-f85039f3317c admin admin] PATH_INFO: `/v3/projects/c8bfec29991445f5aadb0a33910eb92b/users/878e8c69fa8d4aadac3968a602c603c5/roles/c51fe7c5ad804040879acfb54269455f` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:31.492985 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12ea07f8-6ca7-4f1d-8dbf-f85039f3317c admin admin] RBAC: Authorizing `identity:create_grant(project_id=c8bfec29991445f5aadb0a33910eb92b, user_id=878e8c69fa8d4aadac3968a602c603c5, role_id=c51fe7c5ad804040879acfb54269455f)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:31.495576 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12ea07f8-6ca7-4f1d-8dbf-f85039f3317c admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:31.506797 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1322/2619] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:47:31 2022] PUT /identity/v3/projects/c8bfec29991445f5aadb0a33910eb92b/users/878e8c69fa8d4aadac3968a602c603c5/roles/c51fe7c5ad804040879acfb54269455f => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:47:31.518611 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-945990ff-3508-403f-abe8-18d9ce310659 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:31.518941 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-945990ff-3508-403f-abe8-18d9ce310659 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:31.519174 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-945990ff-3508-403f-abe8-18d9ce310659 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:31.550558 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-945990ff-3508-403f-abe8-18d9ce310659 None None] MFA Rules not processed for user `878e8c69fa8d4aadac3968a602c603c5`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:47:31.551787 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-945990ff-3508-403f-abe8-18d9ce310659 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:47:31.626065 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1298/2620] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:47:31 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:47:31.636909 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12ea07f8-6ca7-4f1d-8dbf-f85039f3317c admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:31.638636 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-12ea07f8-6ca7-4f1d-8dbf-f85039f3317c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:47:31.704880 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12ea07f8-6ca7-4f1d-8dbf-f85039f3317c admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:31.710986 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e096d43-9810-45b7-baa0-b4e34a2ba708 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ffbc136c77b84e8c87128f7247f4ec03', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:31.711814 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e096d43-9810-45b7-baa0-b4e34a2ba708 service cinder] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:31.712037 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e096d43-9810-45b7-baa0-b4e34a2ba708 service cinder] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:31.712369 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e096d43-9810-45b7-baa0-b4e34a2ba708 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:31.725472 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e096d43-9810-45b7-baa0-b4e34a2ba708 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:31.726954 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e096d43-9810-45b7-baa0-b4e34a2ba708 service cinder] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:31.737102 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1323/2621] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:47:31 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:47:31.844578 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-945990ff-3508-403f-abe8-18d9ce310659 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:31.854138 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-945990ff-3508-403f-abe8-18d9ce310659 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:31.857665 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9fdb65b-8558-49ee-96ce-2912363c6a9e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ffbc136c77b84e8c87128f7247f4ec03', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:31.858251 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9fdb65b-8558-49ee-96ce-2912363c6a9e service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:31.858530 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9fdb65b-8558-49ee-96ce-2912363c6a9e service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:31.859308 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9fdb65b-8558-49ee-96ce-2912363c6a9e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:31.913633 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:47:31.915898 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9fdb65b-8558-49ee-96ce-2912363c6a9e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:31.917724 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9fdb65b-8558-49ee-96ce-2912363c6a9e service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:31.928207 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1299/2622] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:47:31 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:47:32.824913 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e096d43-9810-45b7-baa0-b4e34a2ba708 service cinder] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:32.826386 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-6e096d43-9810-45b7-baa0-b4e34a2ba708 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:47:32.884495 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e096d43-9810-45b7-baa0-b4e34a2ba708 service cinder] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:32.888042 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10b86e28-b6fe-4df0-9963-14986763a06b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:32.888663 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10b86e28-b6fe-4df0-9963-14986763a06b service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:32.888956 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10b86e28-b6fe-4df0-9963-14986763a06b service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:32.889226 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10b86e28-b6fe-4df0-9963-14986763a06b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:32.892092 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-10b86e28-b6fe-4df0-9963-14986763a06b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:47:32.932475 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:47:32.960913 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10b86e28-b6fe-4df0-9963-14986763a06b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:32.962342 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10b86e28-b6fe-4df0-9963-14986763a06b service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:32.974536 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1324/2623] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:47:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 151 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:47:34.556005 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9fdb65b-8558-49ee-96ce-2912363c6a9e service cinder] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:34.569040 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9fdb65b-8558-49ee-96ce-2912363c6a9e service cinder] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:34.574762 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78302676-14d9-47e9-8e50-94ccb9dec75e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:34.575545 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78302676-14d9-47e9-8e50-94ccb9dec75e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:34.575930 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78302676-14d9-47e9-8e50-94ccb9dec75e admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:34.576275 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78302676-14d9-47e9-8e50-94ccb9dec75e admin admin] PATH_INFO: `/v3/users/d93b21e8ea94486c8527e31b84c635fa` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:34.578466 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78302676-14d9-47e9-8e50-94ccb9dec75e admin admin] RBAC: Authorizing `identity:delete_user(user_id=d93b21e8ea94486c8527e31b84c635fa)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:34.580564 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78302676-14d9-47e9-8e50-94ccb9dec75e admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:34.662072 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-78302676-14d9-47e9-8e50-94ccb9dec75e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd93b21e8ea94486c8527e31b84c635fa'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:34.673707 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-78302676-14d9-47e9-8e50-94ccb9dec75e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd93b21e8ea94486c8527e31b84c635fa'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:34.690299 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-78302676-14d9-47e9-8e50-94ccb9dec75e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd93b21e8ea94486c8527e31b84c635fa'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:34.691049 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-78302676-14d9-47e9-8e50-94ccb9dec75e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd93b21e8ea94486c8527e31b84c635fa'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:34.700793 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-78302676-14d9-47e9-8e50-94ccb9dec75e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd93b21e8ea94486c8527e31b84c635fa'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:34.709146 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1300/2624] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:47:34 2022] DELETE /identity/v3/users/d93b21e8ea94486c8527e31b84c635fa => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 04:47:34.713770 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b967279-1918-4825-a4dd-b2a759e5434d None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:34.714580 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b967279-1918-4825-a4dd-b2a759e5434d None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:34.714580 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b967279-1918-4825-a4dd-b2a759e5434d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:34.740162 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-4b967279-1918-4825-a4dd-b2a759e5434d None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:47:34.741612 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-4b967279-1918-4825-a4dd-b2a759e5434d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:47:34.807773 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1325/2625] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:47:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:47:34.818477 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78302676-14d9-47e9-8e50-94ccb9dec75e admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:34.819946 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-78302676-14d9-47e9-8e50-94ccb9dec75e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:47:34.890658 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78302676-14d9-47e9-8e50-94ccb9dec75e admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:34.895563 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-753f27dc-83d5-46ce-be29-bfae5a139e4b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:34.895844 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-753f27dc-83d5-46ce-be29-bfae5a139e4b service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:34.896131 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-753f27dc-83d5-46ce-be29-bfae5a139e4b service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:34.896411 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-753f27dc-83d5-46ce-be29-bfae5a139e4b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:34.908749 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-753f27dc-83d5-46ce-be29-bfae5a139e4b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:34.910239 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-753f27dc-83d5-46ce-be29-bfae5a139e4b service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:34.922121 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1301/2626] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:47:34 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:47:35.257192 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b967279-1918-4825-a4dd-b2a759e5434d None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:35.259691 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-4b967279-1918-4825-a4dd-b2a759e5434d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:47:35.274004 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b967279-1918-4825-a4dd-b2a759e5434d None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:35.289516 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14c13ce1-421c-4666-b7af-457171bafa46 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:35.290829 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c13ce1-421c-4666-b7af-457171bafa46 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:35.291110 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c13ce1-421c-4666-b7af-457171bafa46 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:35.291377 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c13ce1-421c-4666-b7af-457171bafa46 admin admin] PATH_INFO: `/v3/projects/e17a7ece84f1424cbef9a3a9924b7f93` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:35.293389 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14c13ce1-421c-4666-b7af-457171bafa46 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e17a7ece84f1424cbef9a3a9924b7f93)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:35.295878 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14c13ce1-421c-4666-b7af-457171bafa46 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:35.357124 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-14c13ce1-421c-4666-b7af-457171bafa46 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e17a7ece84f1424cbef9a3a9924b7f93', 'request_id': 'req-14c13ce1-421c-4666-b7af-457171bafa46'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:35.367553 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-14c13ce1-421c-4666-b7af-457171bafa46 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e17a7ece84f1424cbef9a3a9924b7f93', 'request_id': 'req-14c13ce1-421c-4666-b7af-457171bafa46'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:35.374311 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-14c13ce1-421c-4666-b7af-457171bafa46 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e17a7ece84f1424cbef9a3a9924b7f93', 'request_id': 'req-14c13ce1-421c-4666-b7af-457171bafa46'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:35.388792 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-14c13ce1-421c-4666-b7af-457171bafa46 admin admin] The token cache is being invalidate because project e17a7ece84f1424cbef9a3a9924b7f93 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81434) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:47:35.389090 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-14c13ce1-421c-4666-b7af-457171bafa46 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:35.389759 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-14c13ce1-421c-4666-b7af-457171bafa46 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:35.391835 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1326/2627] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:47:35 2022] DELETE /identity/v3/projects/e17a7ece84f1424cbef9a3a9924b7f93 => generated 0 bytes in 135 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:47:35.395859 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-753f27dc-83d5-46ce-be29-bfae5a139e4b service neutron] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:35.408433 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-753f27dc-83d5-46ce-be29-bfae5a139e4b service neutron] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:35.473005 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f57ff3a-4de3-49d9-af14-51f82f51ee63 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:35.473829 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f57ff3a-4de3-49d9-af14-51f82f51ee63 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:35.474354 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f57ff3a-4de3-49d9-af14-51f82f51ee63 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:35.474694 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f57ff3a-4de3-49d9-af14-51f82f51ee63 admin admin] PATH_INFO: `/v3/users/878e8c69fa8d4aadac3968a602c603c5` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:35.475978 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f57ff3a-4de3-49d9-af14-51f82f51ee63 admin admin] RBAC: Authorizing `identity:delete_user(user_id=878e8c69fa8d4aadac3968a602c603c5)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:35.478057 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f57ff3a-4de3-49d9-af14-51f82f51ee63 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:35.494768 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:47:35.538552 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-0f57ff3a-4de3-49d9-af14-51f82f51ee63 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '878e8c69fa8d4aadac3968a602c603c5'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:35.547915 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-0f57ff3a-4de3-49d9-af14-51f82f51ee63 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '878e8c69fa8d4aadac3968a602c603c5'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:35.560217 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-0f57ff3a-4de3-49d9-af14-51f82f51ee63 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '878e8c69fa8d4aadac3968a602c603c5'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:35.561220 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-0f57ff3a-4de3-49d9-af14-51f82f51ee63 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '878e8c69fa8d4aadac3968a602c603c5'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:35.568199 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-0f57ff3a-4de3-49d9-af14-51f82f51ee63 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '878e8c69fa8d4aadac3968a602c603c5'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:35.574079 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1302/2628] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:47:35 2022] DELETE /identity/v3/users/878e8c69fa8d4aadac3968a602c603c5 => generated 0 bytes in 179 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:47:36.216087 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14c13ce1-421c-4666-b7af-457171bafa46 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:36.217993 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-14c13ce1-421c-4666-b7af-457171bafa46 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:47:36.242413 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14c13ce1-421c-4666-b7af-457171bafa46 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:36.325343 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:47:36.336705 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1ad9737-b775-452a-accc-28c2bd271b84 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:36.337938 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1ad9737-b775-452a-accc-28c2bd271b84 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:36.338512 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1ad9737-b775-452a-accc-28c2bd271b84 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:36.338986 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1ad9737-b775-452a-accc-28c2bd271b84 admin admin] PATH_INFO: `/v3/projects/c8bfec29991445f5aadb0a33910eb92b` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:36.341255 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1ad9737-b775-452a-accc-28c2bd271b84 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c8bfec29991445f5aadb0a33910eb92b)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:36.343593 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1ad9737-b775-452a-accc-28c2bd271b84 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:36.398962 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-a1ad9737-b775-452a-accc-28c2bd271b84 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c8bfec29991445f5aadb0a33910eb92b', 'request_id': 'req-a1ad9737-b775-452a-accc-28c2bd271b84'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:36.408300 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-a1ad9737-b775-452a-accc-28c2bd271b84 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c8bfec29991445f5aadb0a33910eb92b', 'request_id': 'req-a1ad9737-b775-452a-accc-28c2bd271b84'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:36.414516 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-a1ad9737-b775-452a-accc-28c2bd271b84 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c8bfec29991445f5aadb0a33910eb92b', 'request_id': 'req-a1ad9737-b775-452a-accc-28c2bd271b84'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:36.430135 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-a1ad9737-b775-452a-accc-28c2bd271b84 admin admin] The token cache is being invalidate because project c8bfec29991445f5aadb0a33910eb92b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81434) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:47:36.430135 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-a1ad9737-b775-452a-accc-28c2bd271b84 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:36.430752 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-a1ad9737-b775-452a-accc-28c2bd271b84 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:36.432721 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1327/2629] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:47:36 2022] DELETE /identity/v3/projects/c8bfec29991445f5aadb0a33910eb92b => generated 0 bytes in 217 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:47:36.437841 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afde693a-540d-46d4-89d0-4b4bded5774b None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:36.438124 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afde693a-540d-46d4-89d0-4b4bded5774b None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:36.438393 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afde693a-540d-46d4-89d0-4b4bded5774b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:36.464558 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-afde693a-540d-46d4-89d0-4b4bded5774b None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:47:36.465739 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-afde693a-540d-46d4-89d0-4b4bded5774b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:47:36.549264 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1303/2630] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:47:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 04:47:36.557114 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a1777b-5573-485c-ae1d-325cf50d75d4 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:36.557899 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a1777b-5573-485c-ae1d-325cf50d75d4 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:36.557899 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a1777b-5573-485c-ae1d-325cf50d75d4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:36.581648 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-36a1777b-5573-485c-ae1d-325cf50d75d4 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:47:36.582877 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-36a1777b-5573-485c-ae1d-325cf50d75d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:47:36.591277 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1328/2631] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:47:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:47:36.596315 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afde693a-540d-46d4-89d0-4b4bded5774b None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:36.606325 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afde693a-540d-46d4-89d0-4b4bded5774b None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:36.611890 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2157075-d440-46a5-9cef-f5b5a0b4ef2f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:36.612519 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2157075-d440-46a5-9cef-f5b5a0b4ef2f admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:36.612844 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2157075-d440-46a5-9cef-f5b5a0b4ef2f admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:36.613121 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2157075-d440-46a5-9cef-f5b5a0b4ef2f admin admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:36.613768 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2157075-d440-46a5-9cef-f5b5a0b4ef2f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:47:36.614074 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2157075-d440-46a5-9cef-f5b5a0b4ef2f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:36.615673 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2157075-d440-46a5-9cef-f5b5a0b4ef2f admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:36.627923 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1304/2632] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Sep 25 04:47:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:47:36.632447 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36a1777b-5573-485c-ae1d-325cf50d75d4 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:36.643608 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36a1777b-5573-485c-ae1d-325cf50d75d4 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:36.651263 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a95335f1-faf0-420a-bd1d-bfb44a1ef7fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:36.651891 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a95335f1-faf0-420a-bd1d-bfb44a1ef7fa admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:36.652175 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a95335f1-faf0-420a-bd1d-bfb44a1ef7fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:36.652500 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a95335f1-faf0-420a-bd1d-bfb44a1ef7fa admin admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:36.653275 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a95335f1-faf0-420a-bd1d-bfb44a1ef7fa admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:36.655240 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a95335f1-faf0-420a-bd1d-bfb44a1ef7fa admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:36.692091 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1329/2633] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:47:36 2022] POST /identity/v3/projects => generated 385 bytes in 60 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:47:36.696764 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2157075-d440-46a5-9cef-f5b5a0b4ef2f admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:36.707021 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2157075-d440-46a5-9cef-f5b5a0b4ef2f admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:36.778777 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f7fe2fa-9f42-4d8c-bd72-f91922cb0fc2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:36.779283 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f7fe2fa-9f42-4d8c-bd72-f91922cb0fc2 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:36.779639 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f7fe2fa-9f42-4d8c-bd72-f91922cb0fc2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:36.779948 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f7fe2fa-9f42-4d8c-bd72-f91922cb0fc2 admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:36.781366 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f7fe2fa-9f42-4d8c-bd72-f91922cb0fc2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:36.783567 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f7fe2fa-9f42-4d8c-bd72-f91922cb0fc2 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:36.807695 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1305/2634] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:47:36 2022] POST /identity/v3/users => generated 293 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 04:47:36.812721 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a95335f1-faf0-420a-bd1d-bfb44a1ef7fa admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:36.826015 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a95335f1-faf0-420a-bd1d-bfb44a1ef7fa admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:36.832016 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d600864a-b236-49af-9970-ecbe6fff51b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:36.832771 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d600864a-b236-49af-9970-ecbe6fff51b9 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:36.833196 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d600864a-b236-49af-9970-ecbe6fff51b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:36.833606 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d600864a-b236-49af-9970-ecbe6fff51b9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:36.834270 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d600864a-b236-49af-9970-ecbe6fff51b9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:36.836018 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d600864a-b236-49af-9970-ecbe6fff51b9 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:36.848223 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1330/2635] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:47:36 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:47:36.852135 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f7fe2fa-9f42-4d8c-bd72-f91922cb0fc2 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:36.863339 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f7fe2fa-9f42-4d8c-bd72-f91922cb0fc2 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:36.868413 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c389e348-a609-4d9b-b3dd-9c6135d2ed07 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:36.869072 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c389e348-a609-4d9b-b3dd-9c6135d2ed07 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:36.869351 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c389e348-a609-4d9b-b3dd-9c6135d2ed07 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:36.869622 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c389e348-a609-4d9b-b3dd-9c6135d2ed07 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:36.870266 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c389e348-a609-4d9b-b3dd-9c6135d2ed07 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:36.871895 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c389e348-a609-4d9b-b3dd-9c6135d2ed07 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:36.873670 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=87,keystone:DELETE=9,keystone:INSERT=4 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:47:36.888202 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1306/2636] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:47:36 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:47:36.889520 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d600864a-b236-49af-9970-ecbe6fff51b9 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:36.900466 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d600864a-b236-49af-9970-ecbe6fff51b9 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:36.911568 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38dfe162-b3db-4a40-8d5b-02a2ec0c457a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:36.912258 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38dfe162-b3db-4a40-8d5b-02a2ec0c457a admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:36.912542 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38dfe162-b3db-4a40-8d5b-02a2ec0c457a admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:36.912946 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38dfe162-b3db-4a40-8d5b-02a2ec0c457a admin admin] PATH_INFO: `/v3/projects/ae42d8eb861e43b4a514d1cb5ec7ec5b/users/30ad4e204c60461ab9cc3ba16b377a21/roles/8f6cc574056c4278bc9778f14dc4b411` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:36.930712 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38dfe162-b3db-4a40-8d5b-02a2ec0c457a admin admin] RBAC: Authorizing `identity:create_grant(project_id=ae42d8eb861e43b4a514d1cb5ec7ec5b, user_id=30ad4e204c60461ab9cc3ba16b377a21, role_id=8f6cc574056c4278bc9778f14dc4b411)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:36.933774 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38dfe162-b3db-4a40-8d5b-02a2ec0c457a admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:36.957191 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1331/2637] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:47:36 2022] PUT /identity/v3/projects/ae42d8eb861e43b4a514d1cb5ec7ec5b/users/30ad4e204c60461ab9cc3ba16b377a21/roles/8f6cc574056c4278bc9778f14dc4b411 => generated 0 bytes in 68 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:47:36.968003 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95ac08b5-9117-4c15-859c-d29b24fb6b99 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:36.968905 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95ac08b5-9117-4c15-859c-d29b24fb6b99 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:36.969275 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95ac08b5-9117-4c15-859c-d29b24fb6b99 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:37.015409 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-95ac08b5-9117-4c15-859c-d29b24fb6b99 None None] MFA Rules not processed for user `30ad4e204c60461ab9cc3ba16b377a21`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:47:37.016840 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-95ac08b5-9117-4c15-859c-d29b24fb6b99 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:47:37.119836 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1307/2638] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:47:36 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:47:37.132823 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38dfe162-b3db-4a40-8d5b-02a2ec0c457a admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:37.135083 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-38dfe162-b3db-4a40-8d5b-02a2ec0c457a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:47:37.210383 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38dfe162-b3db-4a40-8d5b-02a2ec0c457a admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:37.214834 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f406f921-c914-4324-8393-59dd59e10973 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ffbc136c77b84e8c87128f7247f4ec03', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:37.215450 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f406f921-c914-4324-8393-59dd59e10973 service cinder] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:37.215731 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f406f921-c914-4324-8393-59dd59e10973 service cinder] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:37.216005 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f406f921-c914-4324-8393-59dd59e10973 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:37.226862 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f406f921-c914-4324-8393-59dd59e10973 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:37.229026 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f406f921-c914-4324-8393-59dd59e10973 service cinder] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:37.240143 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1332/2639] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:47:37 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:47:38.346030 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95ac08b5-9117-4c15-859c-d29b24fb6b99 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:38.347529 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-95ac08b5-9117-4c15-859c-d29b24fb6b99 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:47:38.414412 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95ac08b5-9117-4c15-859c-d29b24fb6b99 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:38.418955 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61b6a0f4-7d96-4383-9df2-ba1bdec8c634 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:38.419587 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61b6a0f4-7d96-4383-9df2-ba1bdec8c634 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:38.419868 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61b6a0f4-7d96-4383-9df2-ba1bdec8c634 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:38.420144 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61b6a0f4-7d96-4383-9df2-ba1bdec8c634 admin admin] PATH_INFO: `/v3/users/c50ba6f9a164463eae9a248936e7bf80` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:38.422045 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61b6a0f4-7d96-4383-9df2-ba1bdec8c634 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c50ba6f9a164463eae9a248936e7bf80)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:38.424118 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61b6a0f4-7d96-4383-9df2-ba1bdec8c634 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:38.468477 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-61b6a0f4-7d96-4383-9df2-ba1bdec8c634 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c50ba6f9a164463eae9a248936e7bf80'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:38.477517 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-61b6a0f4-7d96-4383-9df2-ba1bdec8c634 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c50ba6f9a164463eae9a248936e7bf80'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:38.490978 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-61b6a0f4-7d96-4383-9df2-ba1bdec8c634 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c50ba6f9a164463eae9a248936e7bf80'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:38.491656 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-61b6a0f4-7d96-4383-9df2-ba1bdec8c634 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c50ba6f9a164463eae9a248936e7bf80'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:38.498371 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-61b6a0f4-7d96-4383-9df2-ba1bdec8c634 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c50ba6f9a164463eae9a248936e7bf80'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:38.505119 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1308/2640] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:47:38 2022] DELETE /identity/v3/users/c50ba6f9a164463eae9a248936e7bf80 => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:47:38.679571 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f406f921-c914-4324-8393-59dd59e10973 service cinder] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:38.696071 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f406f921-c914-4324-8393-59dd59e10973 service cinder] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:38.717495 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:47:38.755648 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4162979-543e-44fb-b4ff-efc048097b48 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:38.756288 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4162979-543e-44fb-b4ff-efc048097b48 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:38.756565 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4162979-543e-44fb-b4ff-efc048097b48 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:38.756875 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4162979-543e-44fb-b4ff-efc048097b48 admin admin] PATH_INFO: `/v3/projects/5ead2221f7f54d6cbea9fc086a089342` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:38.758024 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4162979-543e-44fb-b4ff-efc048097b48 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5ead2221f7f54d6cbea9fc086a089342)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:38.760022 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4162979-543e-44fb-b4ff-efc048097b48 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:38.818017 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-a4162979-543e-44fb-b4ff-efc048097b48 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5ead2221f7f54d6cbea9fc086a089342', 'request_id': 'req-a4162979-543e-44fb-b4ff-efc048097b48'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:38.829189 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-a4162979-543e-44fb-b4ff-efc048097b48 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5ead2221f7f54d6cbea9fc086a089342', 'request_id': 'req-a4162979-543e-44fb-b4ff-efc048097b48'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:38.835846 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-a4162979-543e-44fb-b4ff-efc048097b48 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5ead2221f7f54d6cbea9fc086a089342', 'request_id': 'req-a4162979-543e-44fb-b4ff-efc048097b48'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:38.852861 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-a4162979-543e-44fb-b4ff-efc048097b48 admin admin] The token cache is being invalidate because project 5ead2221f7f54d6cbea9fc086a089342 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81434) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:47:38.853161 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-a4162979-543e-44fb-b4ff-efc048097b48 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:38.853837 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-a4162979-543e-44fb-b4ff-efc048097b48 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:38.856165 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1333/2641] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:47:38 2022] DELETE /identity/v3/projects/5ead2221f7f54d6cbea9fc086a089342 => generated 0 bytes in 177 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:47:38.861856 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95efc997-597a-4d9a-a394-6c9765cb3f75 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:38.862150 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95efc997-597a-4d9a-a394-6c9765cb3f75 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:38.862428 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95efc997-597a-4d9a-a394-6c9765cb3f75 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:38.880297 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:47:38.906336 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-95efc997-597a-4d9a-a394-6c9765cb3f75 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:47:38.907943 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-95efc997-597a-4d9a-a394-6c9765cb3f75 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:47:38.993830 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1309/2642] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:47:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:47:39.002088 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-942c6419-a70c-494f-87b1-5887fcc0de24 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:39.002868 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-942c6419-a70c-494f-87b1-5887fcc0de24 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:39.003415 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-942c6419-a70c-494f-87b1-5887fcc0de24 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:39.028570 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-942c6419-a70c-494f-87b1-5887fcc0de24 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:47:39.030071 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-942c6419-a70c-494f-87b1-5887fcc0de24 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:47:39.039328 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1334/2643] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:47:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:47:39.043901 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95efc997-597a-4d9a-a394-6c9765cb3f75 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:39.059252 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95efc997-597a-4d9a-a394-6c9765cb3f75 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:39.065435 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a345bb8f-72b4-42ec-8eb8-2476828cf483 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:39.066242 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a345bb8f-72b4-42ec-8eb8-2476828cf483 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:39.066530 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a345bb8f-72b4-42ec-8eb8-2476828cf483 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:39.066799 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a345bb8f-72b4-42ec-8eb8-2476828cf483 admin admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:39.068143 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a345bb8f-72b4-42ec-8eb8-2476828cf483 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:47:39.068143 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a345bb8f-72b4-42ec-8eb8-2476828cf483 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:39.069430 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a345bb8f-72b4-42ec-8eb8-2476828cf483 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:39.084551 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1310/2644] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Sep 25 04:47:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:47:39.088605 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-942c6419-a70c-494f-87b1-5887fcc0de24 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:39.098237 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-942c6419-a70c-494f-87b1-5887fcc0de24 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:39.102945 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d040e15d-060e-4901-9c2f-f1d80c822ffd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:39.103548 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d040e15d-060e-4901-9c2f-f1d80c822ffd admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:39.103827 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d040e15d-060e-4901-9c2f-f1d80c822ffd admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:39.104096 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d040e15d-060e-4901-9c2f-f1d80c822ffd admin admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:39.104753 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d040e15d-060e-4901-9c2f-f1d80c822ffd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:39.106679 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d040e15d-060e-4901-9c2f-f1d80c822ffd admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:39.134662 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1335/2645] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:47:39 2022] POST /identity/v3/projects => generated 375 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:47:39.139015 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a345bb8f-72b4-42ec-8eb8-2476828cf483 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:39.150484 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a345bb8f-72b4-42ec-8eb8-2476828cf483 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:39.215371 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbbb6ceb-16a4-4a90-9401-51ec860a9df7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:39.216023 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbbb6ceb-16a4-4a90-9401-51ec860a9df7 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:39.216311 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbbb6ceb-16a4-4a90-9401-51ec860a9df7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:39.216590 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbbb6ceb-16a4-4a90-9401-51ec860a9df7 admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:39.217296 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbbb6ceb-16a4-4a90-9401-51ec860a9df7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:39.219275 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbbb6ceb-16a4-4a90-9401-51ec860a9df7 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:39.245083 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1311/2646] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:47:39 2022] POST /identity/v3/users => generated 288 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:47:39.246925 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d040e15d-060e-4901-9c2f-f1d80c822ffd admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:39.262646 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d040e15d-060e-4901-9c2f-f1d80c822ffd admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:39.267442 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebd34ed5-5878-4770-842c-34cdc8f94039 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:39.268075 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebd34ed5-5878-4770-842c-34cdc8f94039 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:39.268348 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebd34ed5-5878-4770-842c-34cdc8f94039 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:39.268679 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebd34ed5-5878-4770-842c-34cdc8f94039 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:39.269322 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebd34ed5-5878-4770-842c-34cdc8f94039 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:39.270901 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebd34ed5-5878-4770-842c-34cdc8f94039 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:39.281743 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1336/2647] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:47:39 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:47:39.285832 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbbb6ceb-16a4-4a90-9401-51ec860a9df7 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:39.296841 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbbb6ceb-16a4-4a90-9401-51ec860a9df7 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:39.301615 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a07519d9-affb-4ac0-9f8c-792415567fa1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:39.302251 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a07519d9-affb-4ac0-9f8c-792415567fa1 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:39.302537 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a07519d9-affb-4ac0-9f8c-792415567fa1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:39.302814 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a07519d9-affb-4ac0-9f8c-792415567fa1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:39.303461 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a07519d9-affb-4ac0-9f8c-792415567fa1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:39.305110 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a07519d9-affb-4ac0-9f8c-792415567fa1 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:39.321767 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1312/2648] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:47:39 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:47:39.323225 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebd34ed5-5878-4770-842c-34cdc8f94039 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:39.333535 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebd34ed5-5878-4770-842c-34cdc8f94039 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:39.338461 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee0f113b-1759-4364-bfe9-2fb260aa5449 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:39.339360 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee0f113b-1759-4364-bfe9-2fb260aa5449 admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:39.339651 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee0f113b-1759-4364-bfe9-2fb260aa5449 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:39.340042 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee0f113b-1759-4364-bfe9-2fb260aa5449 admin admin] PATH_INFO: `/v3/projects/e01ea8b16a414e6a861fd1fee716228a/users/7cbd34954b8444a7aaa8d12ebd2ff4ca/roles/8f6cc574056c4278bc9778f14dc4b411` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:39.360118 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee0f113b-1759-4364-bfe9-2fb260aa5449 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e01ea8b16a414e6a861fd1fee716228a, user_id=7cbd34954b8444a7aaa8d12ebd2ff4ca, role_id=8f6cc574056c4278bc9778f14dc4b411)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:39.362023 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee0f113b-1759-4364-bfe9-2fb260aa5449 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:39.376774 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1337/2649] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:47:39 2022] PUT /identity/v3/projects/e01ea8b16a414e6a861fd1fee716228a/users/7cbd34954b8444a7aaa8d12ebd2ff4ca/roles/8f6cc574056c4278bc9778f14dc4b411 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:47:39.388334 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b914e130-2730-4f15-a158-41911164c4a9 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:39.388894 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b914e130-2730-4f15-a158-41911164c4a9 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:39.388894 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b914e130-2730-4f15-a158-41911164c4a9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:39.417726 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-b914e130-2730-4f15-a158-41911164c4a9 None None] MFA Rules not processed for user `7cbd34954b8444a7aaa8d12ebd2ff4ca`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:47:39.418983 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-b914e130-2730-4f15-a158-41911164c4a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:47:39.486169 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:47:39.490182 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1313/2650] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:47:39 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:47:39.501548 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee0f113b-1759-4364-bfe9-2fb260aa5449 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:39.503346 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-ee0f113b-1759-4364-bfe9-2fb260aa5449 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:47:39.556091 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:47:39.578827 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee0f113b-1759-4364-bfe9-2fb260aa5449 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:39.584407 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-622c2b3a-8aa4-4fc3-8fcc-b3abe26fdce0 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e703b42c53974daaa608b0ccc3f018b1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'ResellerAdmin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:39.585051 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-622c2b3a-8aa4-4fc3-8fcc-b3abe26fdce0 service nova] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:39.585333 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-622c2b3a-8aa4-4fc3-8fcc-b3abe26fdce0 service nova] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:39.585608 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-622c2b3a-8aa4-4fc3-8fcc-b3abe26fdce0 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:39.594228 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-622c2b3a-8aa4-4fc3-8fcc-b3abe26fdce0 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:39.595829 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-622c2b3a-8aa4-4fc3-8fcc-b3abe26fdce0 service nova] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:39.604558 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1338/2651] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:47:39 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:47:39.817223 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b914e130-2730-4f15-a158-41911164c4a9 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:39.879511 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b914e130-2730-4f15-a158-41911164c4a9 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:39.884076 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e2c0b3b-5ebc-48ac-b41f-c86f59ea9293 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:39.884737 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e2c0b3b-5ebc-48ac-b41f-c86f59ea9293 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:39.885024 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e2c0b3b-5ebc-48ac-b41f-c86f59ea9293 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:39.885300 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e2c0b3b-5ebc-48ac-b41f-c86f59ea9293 admin admin] PATH_INFO: `/v3/users/7cbd34954b8444a7aaa8d12ebd2ff4ca` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:39.886436 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e2c0b3b-5ebc-48ac-b41f-c86f59ea9293 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7cbd34954b8444a7aaa8d12ebd2ff4ca)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:39.888355 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e2c0b3b-5ebc-48ac-b41f-c86f59ea9293 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:39.937693 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-8e2c0b3b-5ebc-48ac-b41f-c86f59ea9293 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7cbd34954b8444a7aaa8d12ebd2ff4ca'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:39.944746 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-8e2c0b3b-5ebc-48ac-b41f-c86f59ea9293 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7cbd34954b8444a7aaa8d12ebd2ff4ca'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:39.957512 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-8e2c0b3b-5ebc-48ac-b41f-c86f59ea9293 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7cbd34954b8444a7aaa8d12ebd2ff4ca'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:39.958154 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-8e2c0b3b-5ebc-48ac-b41f-c86f59ea9293 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7cbd34954b8444a7aaa8d12ebd2ff4ca'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:39.965273 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-8e2c0b3b-5ebc-48ac-b41f-c86f59ea9293 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7cbd34954b8444a7aaa8d12ebd2ff4ca'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:39.971795 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1314/2652] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:47:39 2022] DELETE /identity/v3/users/7cbd34954b8444a7aaa8d12ebd2ff4ca => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:47:39.976191 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3d605ee-db49-46ed-8dfd-6c8e45203c35 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:39.976700 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3d605ee-db49-46ed-8dfd-6c8e45203c35 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:39.976803 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3d605ee-db49-46ed-8dfd-6c8e45203c35 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:39.995242 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-d3d605ee-db49-46ed-8dfd-6c8e45203c35 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:47:39.996429 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-d3d605ee-db49-46ed-8dfd-6c8e45203c35 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:47:40.058064 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1339/2653] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:47:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:47:40.067793 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e2c0b3b-5ebc-48ac-b41f-c86f59ea9293 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:40.069255 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-8e2c0b3b-5ebc-48ac-b41f-c86f59ea9293 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:47:40.126737 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e2c0b3b-5ebc-48ac-b41f-c86f59ea9293 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:40.130431 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8e06ad8-7519-40e6-84da-bd27b40d84a4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:40.131033 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8e06ad8-7519-40e6-84da-bd27b40d84a4 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:40.131316 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8e06ad8-7519-40e6-84da-bd27b40d84a4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:40.131590 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8e06ad8-7519-40e6-84da-bd27b40d84a4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:40.141231 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8e06ad8-7519-40e6-84da-bd27b40d84a4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:40.142663 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8e06ad8-7519-40e6-84da-bd27b40d84a4 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:40.151962 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1315/2654] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:47:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:47:40.422008 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3d605ee-db49-46ed-8dfd-6c8e45203c35 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:40.423648 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-d3d605ee-db49-46ed-8dfd-6c8e45203c35 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:47:40.439758 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3d605ee-db49-46ed-8dfd-6c8e45203c35 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:40.444459 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef18acc3-91a6-4974-88b5-7a9892e894ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:40.445237 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef18acc3-91a6-4974-88b5-7a9892e894ec admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:40.445412 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef18acc3-91a6-4974-88b5-7a9892e894ec admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:40.445682 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef18acc3-91a6-4974-88b5-7a9892e894ec admin admin] PATH_INFO: `/v3/projects/e01ea8b16a414e6a861fd1fee716228a` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:40.446792 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef18acc3-91a6-4974-88b5-7a9892e894ec admin admin] RBAC: Authorizing `identity:delete_project(project_id=e01ea8b16a414e6a861fd1fee716228a)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:40.448827 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef18acc3-91a6-4974-88b5-7a9892e894ec admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:40.494934 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-ef18acc3-91a6-4974-88b5-7a9892e894ec admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e01ea8b16a414e6a861fd1fee716228a', 'request_id': 'req-ef18acc3-91a6-4974-88b5-7a9892e894ec'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:40.502653 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-ef18acc3-91a6-4974-88b5-7a9892e894ec admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e01ea8b16a414e6a861fd1fee716228a', 'request_id': 'req-ef18acc3-91a6-4974-88b5-7a9892e894ec'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:40.508161 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-ef18acc3-91a6-4974-88b5-7a9892e894ec admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e01ea8b16a414e6a861fd1fee716228a', 'request_id': 'req-ef18acc3-91a6-4974-88b5-7a9892e894ec'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:40.520159 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-ef18acc3-91a6-4974-88b5-7a9892e894ec admin admin] The token cache is being invalidate because project e01ea8b16a414e6a861fd1fee716228a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81434) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:47:40.520454 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-ef18acc3-91a6-4974-88b5-7a9892e894ec admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:40.521151 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-ef18acc3-91a6-4974-88b5-7a9892e894ec admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:40.522627 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1340/2655] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:47:40 2022] DELETE /identity/v3/projects/e01ea8b16a414e6a861fd1fee716228a => generated 0 bytes in 101 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:47:40.527445 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e93beb28-0149-402b-b15c-9b3eddac653a None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:40.527968 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e93beb28-0149-402b-b15c-9b3eddac653a None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:40.527968 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e93beb28-0149-402b-b15c-9b3eddac653a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:40.548128 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-e93beb28-0149-402b-b15c-9b3eddac653a None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:47:40.549369 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-e93beb28-0149-402b-b15c-9b3eddac653a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:47:40.566967 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:47:40.611909 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1316/2656] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:47:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:47:40.619435 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-005c1589-20e8-4cf3-bbf1-dd232729973d None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:40.619721 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-005c1589-20e8-4cf3-bbf1-dd232729973d None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:40.620025 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-005c1589-20e8-4cf3-bbf1-dd232729973d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:40.639835 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-005c1589-20e8-4cf3-bbf1-dd232729973d None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:47:40.641070 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-005c1589-20e8-4cf3-bbf1-dd232729973d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:47:40.648991 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1341/2657] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:47:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:47:40.653054 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e93beb28-0149-402b-b15c-9b3eddac653a None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:40.662636 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e93beb28-0149-402b-b15c-9b3eddac653a None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:40.667236 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52f095b1-7dae-4e68-9af8-a0b19db33f2b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:40.667826 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52f095b1-7dae-4e68-9af8-a0b19db33f2b admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:40.668105 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52f095b1-7dae-4e68-9af8-a0b19db33f2b admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:40.668373 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52f095b1-7dae-4e68-9af8-a0b19db33f2b admin admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:40.669040 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52f095b1-7dae-4e68-9af8-a0b19db33f2b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:47:40.669344 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52f095b1-7dae-4e68-9af8-a0b19db33f2b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:40.670927 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52f095b1-7dae-4e68-9af8-a0b19db33f2b admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:40.682692 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1317/2658] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Sep 25 04:47:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:47:40.687536 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-005c1589-20e8-4cf3-bbf1-dd232729973d None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:40.696553 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-005c1589-20e8-4cf3-bbf1-dd232729973d None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:40.701222 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86791ac7-4349-4cc7-90d1-12968f68f47d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:40.701812 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86791ac7-4349-4cc7-90d1-12968f68f47d admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:40.702091 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86791ac7-4349-4cc7-90d1-12968f68f47d admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:40.702390 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86791ac7-4349-4cc7-90d1-12968f68f47d admin admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:40.703010 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86791ac7-4349-4cc7-90d1-12968f68f47d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:40.704942 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86791ac7-4349-4cc7-90d1-12968f68f47d admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:40.731552 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1342/2659] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:47:40 2022] POST /identity/v3/projects => generated 361 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:47:40.736194 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52f095b1-7dae-4e68-9af8-a0b19db33f2b admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:40.746496 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52f095b1-7dae-4e68-9af8-a0b19db33f2b admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:40.801695 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4d57f75-e092-400b-ba75-6ce98f455d47 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:40.802307 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4d57f75-e092-400b-ba75-6ce98f455d47 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:40.802591 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4d57f75-e092-400b-ba75-6ce98f455d47 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:40.802867 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4d57f75-e092-400b-ba75-6ce98f455d47 admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:40.803507 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4d57f75-e092-400b-ba75-6ce98f455d47 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:40.805498 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4d57f75-e092-400b-ba75-6ce98f455d47 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:40.828071 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1318/2660] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:47:40 2022] POST /identity/v3/users => generated 281 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:47:40.832178 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86791ac7-4349-4cc7-90d1-12968f68f47d admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:40.842646 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86791ac7-4349-4cc7-90d1-12968f68f47d admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:40.847516 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf1ff1fe-0ae9-4bfd-8f85-23ea19067910 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:40.848115 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf1ff1fe-0ae9-4bfd-8f85-23ea19067910 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:40.848393 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf1ff1fe-0ae9-4bfd-8f85-23ea19067910 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:40.848691 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf1ff1fe-0ae9-4bfd-8f85-23ea19067910 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:40.849344 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf1ff1fe-0ae9-4bfd-8f85-23ea19067910 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:40.850947 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf1ff1fe-0ae9-4bfd-8f85-23ea19067910 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:40.861674 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1343/2661] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:47:40 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:47:40.865562 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4d57f75-e092-400b-ba75-6ce98f455d47 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:40.876174 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4d57f75-e092-400b-ba75-6ce98f455d47 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:40.880901 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb062c16-ec0c-4c8f-989e-7d20b3cd32cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:40.881490 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb062c16-ec0c-4c8f-989e-7d20b3cd32cc admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:40.881772 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb062c16-ec0c-4c8f-989e-7d20b3cd32cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:40.882047 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb062c16-ec0c-4c8f-989e-7d20b3cd32cc admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:40.882681 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb062c16-ec0c-4c8f-989e-7d20b3cd32cc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:40.884268 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb062c16-ec0c-4c8f-989e-7d20b3cd32cc admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:40.895172 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1319/2662] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:47:40 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:47:40.899342 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf1ff1fe-0ae9-4bfd-8f85-23ea19067910 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:40.909969 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf1ff1fe-0ae9-4bfd-8f85-23ea19067910 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:40.914522 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8953d940-908f-4ec0-82b5-67b476b5ed8a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:40.915163 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8953d940-908f-4ec0-82b5-67b476b5ed8a admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:40.915439 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8953d940-908f-4ec0-82b5-67b476b5ed8a admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:40.915711 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8953d940-908f-4ec0-82b5-67b476b5ed8a admin admin] PATH_INFO: `/v3/projects/e372a081e96740d093fb34612aeaf3ce/users/847208664b224ae7923d8387771e47f3/roles/8f6cc574056c4278bc9778f14dc4b411` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:40.930683 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8953d940-908f-4ec0-82b5-67b476b5ed8a admin admin] RBAC: Authorizing `identity:create_grant(project_id=e372a081e96740d093fb34612aeaf3ce, user_id=847208664b224ae7923d8387771e47f3, role_id=8f6cc574056c4278bc9778f14dc4b411)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:40.933331 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8953d940-908f-4ec0-82b5-67b476b5ed8a admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:40.941668 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:47:40.947939 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1344/2663] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:47:40 2022] PUT /identity/v3/projects/e372a081e96740d093fb34612aeaf3ce/users/847208664b224ae7923d8387771e47f3/roles/8f6cc574056c4278bc9778f14dc4b411 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:47:40.958649 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d41e90-f3e2-4ca1-ae55-7ce480d25fc5 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:40.959073 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d41e90-f3e2-4ca1-ae55-7ce480d25fc5 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:40.959261 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d41e90-f3e2-4ca1-ae55-7ce480d25fc5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:40.994391 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-e6d41e90-f3e2-4ca1-ae55-7ce480d25fc5 None None] MFA Rules not processed for user `847208664b224ae7923d8387771e47f3`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:47:40.995626 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-e6d41e90-f3e2-4ca1-ae55-7ce480d25fc5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:47:41.058067 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1320/2664] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:47:40 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:47:41.062895 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8953d940-908f-4ec0-82b5-67b476b5ed8a admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:41.131484 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8953d940-908f-4ec0-82b5-67b476b5ed8a admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:41.136214 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-223c69fa-8731-471d-b0b4-8c2ca701577c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:41.136861 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-223c69fa-8731-471d-b0b4-8c2ca701577c admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:41.137148 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-223c69fa-8731-471d-b0b4-8c2ca701577c admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:41.137421 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-223c69fa-8731-471d-b0b4-8c2ca701577c admin admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:41.138066 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-223c69fa-8731-471d-b0b4-8c2ca701577c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:41.140100 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-223c69fa-8731-471d-b0b4-8c2ca701577c admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:41.166892 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1345/2665] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:47:41 2022] POST /identity/v3/projects => generated 365 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 04:47:41.170998 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6d41e90-f3e2-4ca1-ae55-7ce480d25fc5 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:41.182620 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6d41e90-f3e2-4ca1-ae55-7ce480d25fc5 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:41.195054 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:47:41.241411 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7575e259-ea71-43ad-a0fc-2419da8127fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:41.242117 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7575e259-ea71-43ad-a0fc-2419da8127fa admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:41.242426 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7575e259-ea71-43ad-a0fc-2419da8127fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:41.242764 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7575e259-ea71-43ad-a0fc-2419da8127fa admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:41.243517 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7575e259-ea71-43ad-a0fc-2419da8127fa admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:41.245883 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7575e259-ea71-43ad-a0fc-2419da8127fa admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:41.267577 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1321/2666] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:47:41 2022] POST /identity/v3/users => generated 289 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:47:41.271581 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-223c69fa-8731-471d-b0b4-8c2ca701577c admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:41.282632 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-223c69fa-8731-471d-b0b4-8c2ca701577c admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:41.287228 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0e05d40-8d72-4603-882c-b96170183c75 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:41.287829 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e05d40-8d72-4603-882c-b96170183c75 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:41.288108 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e05d40-8d72-4603-882c-b96170183c75 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:41.288381 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e05d40-8d72-4603-882c-b96170183c75 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:41.289047 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0e05d40-8d72-4603-882c-b96170183c75 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:41.290621 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0e05d40-8d72-4603-882c-b96170183c75 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:41.301053 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1346/2667] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:47:41 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:47:41.305019 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7575e259-ea71-43ad-a0fc-2419da8127fa admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:41.314859 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7575e259-ea71-43ad-a0fc-2419da8127fa admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:41.319500 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bd48634-e629-4bca-8093-ef0fecf11123 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:41.320150 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bd48634-e629-4bca-8093-ef0fecf11123 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:41.320429 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bd48634-e629-4bca-8093-ef0fecf11123 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:41.320731 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bd48634-e629-4bca-8093-ef0fecf11123 admin admin] PATH_INFO: `/v3/projects/f98c054b48064046a58f95bb28d7b4e1/users/7f80704e73654820a0b6342b71699aa7/roles/c51fe7c5ad804040879acfb54269455f` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:41.336039 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bd48634-e629-4bca-8093-ef0fecf11123 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f98c054b48064046a58f95bb28d7b4e1, user_id=7f80704e73654820a0b6342b71699aa7, role_id=c51fe7c5ad804040879acfb54269455f)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:41.338770 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bd48634-e629-4bca-8093-ef0fecf11123 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:41.352178 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1322/2668] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:47:41 2022] PUT /identity/v3/projects/f98c054b48064046a58f95bb28d7b4e1/users/7f80704e73654820a0b6342b71699aa7/roles/c51fe7c5ad804040879acfb54269455f => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:47:41.356073 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0e05d40-8d72-4603-882c-b96170183c75 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:41.366115 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0e05d40-8d72-4603-882c-b96170183c75 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:41.422491 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f84322a3-803d-4905-83a3-8def6c0a2ab7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:41.423095 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f84322a3-803d-4905-83a3-8def6c0a2ab7 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:41.423371 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f84322a3-803d-4905-83a3-8def6c0a2ab7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:41.423642 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f84322a3-803d-4905-83a3-8def6c0a2ab7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:41.424275 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f84322a3-803d-4905-83a3-8def6c0a2ab7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:41.426214 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f84322a3-803d-4905-83a3-8def6c0a2ab7 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:41.435607 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1347/2669] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:47:41 2022] GET /identity/v3/roles => generated 1420 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:47:41.440037 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bd48634-e629-4bca-8093-ef0fecf11123 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:41.450209 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bd48634-e629-4bca-8093-ef0fecf11123 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:41.455685 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e51f3423-9670-47af-9464-324a05ac6cde admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:41.456326 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e51f3423-9670-47af-9464-324a05ac6cde admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:41.456605 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e51f3423-9670-47af-9464-324a05ac6cde admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:41.456910 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e51f3423-9670-47af-9464-324a05ac6cde admin admin] PATH_INFO: `/v3/projects/f98c054b48064046a58f95bb28d7b4e1/users/7f80704e73654820a0b6342b71699aa7/roles/c51fe7c5ad804040879acfb54269455f` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:41.458877 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e51f3423-9670-47af-9464-324a05ac6cde admin admin] RBAC: Authorizing `identity:create_grant(project_id=f98c054b48064046a58f95bb28d7b4e1, user_id=7f80704e73654820a0b6342b71699aa7, role_id=c51fe7c5ad804040879acfb54269455f)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:41.461515 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e51f3423-9670-47af-9464-324a05ac6cde admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:41.473752 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1323/2670] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:47:41 2022] PUT /identity/v3/projects/f98c054b48064046a58f95bb28d7b4e1/users/7f80704e73654820a0b6342b71699aa7/roles/c51fe7c5ad804040879acfb54269455f => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:47:41.485290 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f29e2520-5e22-42ea-8407-ab695860318b None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:41.485568 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f29e2520-5e22-42ea-8407-ab695860318b None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:41.485835 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f29e2520-5e22-42ea-8407-ab695860318b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:41.512932 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-f29e2520-5e22-42ea-8407-ab695860318b None None] MFA Rules not processed for user `7f80704e73654820a0b6342b71699aa7`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:47:41.514121 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-f29e2520-5e22-42ea-8407-ab695860318b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:47:41.570461 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:47:41.582682 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1348/2671] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:47:41 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:47:41.595418 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e51f3423-9670-47af-9464-324a05ac6cde admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:41.597224 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-e51f3423-9670-47af-9464-324a05ac6cde admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:47:41.655472 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e51f3423-9670-47af-9464-324a05ac6cde admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:41.659444 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e762ed1d-59a8-4d52-9230-6dea1c658c9a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ffbc136c77b84e8c87128f7247f4ec03', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:41.660054 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e762ed1d-59a8-4d52-9230-6dea1c658c9a service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:41.660337 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e762ed1d-59a8-4d52-9230-6dea1c658c9a service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:41.660613 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e762ed1d-59a8-4d52-9230-6dea1c658c9a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:41.669646 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e762ed1d-59a8-4d52-9230-6dea1c658c9a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:41.671171 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e762ed1d-59a8-4d52-9230-6dea1c658c9a service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:41.680696 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1324/2672] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:47:41 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:47:42.105864 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f29e2520-5e22-42ea-8407-ab695860318b None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:42.166942 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f29e2520-5e22-42ea-8407-ab695860318b None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:42.171519 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e86cc008-5de3-4dce-b8fb-9201277fb12b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:42.172148 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e86cc008-5de3-4dce-b8fb-9201277fb12b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:42.172428 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e86cc008-5de3-4dce-b8fb-9201277fb12b admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:42.172734 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e86cc008-5de3-4dce-b8fb-9201277fb12b admin admin] PATH_INFO: `/v3/users/847208664b224ae7923d8387771e47f3` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:42.173862 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e86cc008-5de3-4dce-b8fb-9201277fb12b admin admin] RBAC: Authorizing `identity:delete_user(user_id=847208664b224ae7923d8387771e47f3)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:42.175783 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e86cc008-5de3-4dce-b8fb-9201277fb12b admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:42.223250 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-e86cc008-5de3-4dce-b8fb-9201277fb12b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '847208664b224ae7923d8387771e47f3'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:42.230827 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-e86cc008-5de3-4dce-b8fb-9201277fb12b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '847208664b224ae7923d8387771e47f3'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:42.243457 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-e86cc008-5de3-4dce-b8fb-9201277fb12b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '847208664b224ae7923d8387771e47f3'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:42.244332 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-e86cc008-5de3-4dce-b8fb-9201277fb12b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '847208664b224ae7923d8387771e47f3'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:42.251591 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-e86cc008-5de3-4dce-b8fb-9201277fb12b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '847208664b224ae7923d8387771e47f3'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:42.257382 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1349/2673] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:47:42 2022] DELETE /identity/v3/users/847208664b224ae7923d8387771e47f3 => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:47:42.261940 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-315bbc3e-ca65-43fb-b2d2-e085e616eb93 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:42.262223 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-315bbc3e-ca65-43fb-b2d2-e085e616eb93 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:42.262493 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-315bbc3e-ca65-43fb-b2d2-e085e616eb93 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:42.282778 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-315bbc3e-ca65-43fb-b2d2-e085e616eb93 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:47:42.283968 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-315bbc3e-ca65-43fb-b2d2-e085e616eb93 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:47:42.341565 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1325/2674] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:47:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:47:42.352581 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e86cc008-5de3-4dce-b8fb-9201277fb12b admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:42.354023 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-e86cc008-5de3-4dce-b8fb-9201277fb12b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:47:42.412571 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e86cc008-5de3-4dce-b8fb-9201277fb12b admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:42.416397 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b2b6df4-1d99-472a-8c37-c6eba7c47db9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:42.417060 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b2b6df4-1d99-472a-8c37-c6eba7c47db9 service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:42.417344 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b2b6df4-1d99-472a-8c37-c6eba7c47db9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:42.417619 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b2b6df4-1d99-472a-8c37-c6eba7c47db9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:42.427184 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b2b6df4-1d99-472a-8c37-c6eba7c47db9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:42.428601 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b2b6df4-1d99-472a-8c37-c6eba7c47db9 service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:42.438321 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1350/2675] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:47:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:47:42.714547 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-315bbc3e-ca65-43fb-b2d2-e085e616eb93 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:42.716289 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-315bbc3e-ca65-43fb-b2d2-e085e616eb93 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:47:42.738236 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-315bbc3e-ca65-43fb-b2d2-e085e616eb93 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:42.743180 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f00077d9-545b-4b22-a90c-06de24b0d76c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:42.743829 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f00077d9-545b-4b22-a90c-06de24b0d76c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:42.744115 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f00077d9-545b-4b22-a90c-06de24b0d76c admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:42.744404 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f00077d9-545b-4b22-a90c-06de24b0d76c admin admin] PATH_INFO: `/v3/projects/e372a081e96740d093fb34612aeaf3ce` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:42.745587 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f00077d9-545b-4b22-a90c-06de24b0d76c admin admin] RBAC: Authorizing `identity:delete_project(project_id=e372a081e96740d093fb34612aeaf3ce)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:42.747933 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f00077d9-545b-4b22-a90c-06de24b0d76c admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:42.757783 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:47:42.798970 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-f00077d9-545b-4b22-a90c-06de24b0d76c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e372a081e96740d093fb34612aeaf3ce', 'request_id': 'req-f00077d9-545b-4b22-a90c-06de24b0d76c'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:42.806269 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-f00077d9-545b-4b22-a90c-06de24b0d76c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e372a081e96740d093fb34612aeaf3ce', 'request_id': 'req-f00077d9-545b-4b22-a90c-06de24b0d76c'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:42.811756 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-f00077d9-545b-4b22-a90c-06de24b0d76c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e372a081e96740d093fb34612aeaf3ce', 'request_id': 'req-f00077d9-545b-4b22-a90c-06de24b0d76c'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:42.824163 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-f00077d9-545b-4b22-a90c-06de24b0d76c admin admin] The token cache is being invalidate because project e372a081e96740d093fb34612aeaf3ce was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:47:42.824478 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-f00077d9-545b-4b22-a90c-06de24b0d76c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:42.825162 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-f00077d9-545b-4b22-a90c-06de24b0d76c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:42.826655 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1326/2676] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:47:42 2022] DELETE /identity/v3/projects/e372a081e96740d093fb34612aeaf3ce => generated 0 bytes in 113 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:47:42.831351 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b2b6df4-1d99-472a-8c37-c6eba7c47db9 service neutron] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:42.841473 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b2b6df4-1d99-472a-8c37-c6eba7c47db9 service neutron] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:42.853842 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:47:42.900238 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d123be0-ff5d-4da3-8340-ca4749f4b160 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:42.900908 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d123be0-ff5d-4da3-8340-ca4749f4b160 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:42.901190 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d123be0-ff5d-4da3-8340-ca4749f4b160 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:42.901458 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d123be0-ff5d-4da3-8340-ca4749f4b160 admin admin] PATH_INFO: `/v3/users/7f80704e73654820a0b6342b71699aa7` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:42.902715 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d123be0-ff5d-4da3-8340-ca4749f4b160 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7f80704e73654820a0b6342b71699aa7)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:42.904720 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d123be0-ff5d-4da3-8340-ca4749f4b160 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:42.949143 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-4d123be0-ff5d-4da3-8340-ca4749f4b160 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7f80704e73654820a0b6342b71699aa7'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:42.957372 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-4d123be0-ff5d-4da3-8340-ca4749f4b160 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7f80704e73654820a0b6342b71699aa7'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:42.969867 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-4d123be0-ff5d-4da3-8340-ca4749f4b160 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7f80704e73654820a0b6342b71699aa7'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:42.970499 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-4d123be0-ff5d-4da3-8340-ca4749f4b160 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7f80704e73654820a0b6342b71699aa7'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:42.977307 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-4d123be0-ff5d-4da3-8340-ca4749f4b160 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7f80704e73654820a0b6342b71699aa7'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:42.983142 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1351/2677] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:47:42 2022] DELETE /identity/v3/users/7f80704e73654820a0b6342b71699aa7 => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 04:47:43.257541 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f00077d9-545b-4b22-a90c-06de24b0d76c admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:43.259005 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-f00077d9-545b-4b22-a90c-06de24b0d76c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:47:43.273728 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f00077d9-545b-4b22-a90c-06de24b0d76c admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:43.334432 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2fc3654-14d4-41df-a894-3ad0d22f23ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:43.335102 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2fc3654-14d4-41df-a894-3ad0d22f23ce admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:43.335386 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2fc3654-14d4-41df-a894-3ad0d22f23ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:43.335659 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2fc3654-14d4-41df-a894-3ad0d22f23ce admin admin] PATH_INFO: `/v3/projects/f98c054b48064046a58f95bb28d7b4e1` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:43.336811 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2fc3654-14d4-41df-a894-3ad0d22f23ce admin admin] RBAC: Authorizing `identity:delete_project(project_id=f98c054b48064046a58f95bb28d7b4e1)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:43.338718 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2fc3654-14d4-41df-a894-3ad0d22f23ce admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:43.386882 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-e2fc3654-14d4-41df-a894-3ad0d22f23ce admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f98c054b48064046a58f95bb28d7b4e1', 'request_id': 'req-e2fc3654-14d4-41df-a894-3ad0d22f23ce'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:43.393703 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-e2fc3654-14d4-41df-a894-3ad0d22f23ce admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f98c054b48064046a58f95bb28d7b4e1', 'request_id': 'req-e2fc3654-14d4-41df-a894-3ad0d22f23ce'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:43.400243 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-e2fc3654-14d4-41df-a894-3ad0d22f23ce admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f98c054b48064046a58f95bb28d7b4e1', 'request_id': 'req-e2fc3654-14d4-41df-a894-3ad0d22f23ce'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:43.413467 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-e2fc3654-14d4-41df-a894-3ad0d22f23ce admin admin] The token cache is being invalidate because project f98c054b48064046a58f95bb28d7b4e1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:47:43.413702 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-e2fc3654-14d4-41df-a894-3ad0d22f23ce admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:43.414511 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-e2fc3654-14d4-41df-a894-3ad0d22f23ce admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:43.416350 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1327/2678] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:47:43 2022] DELETE /identity/v3/projects/f98c054b48064046a58f95bb28d7b4e1 => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 04:47:43.423150 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a52d1da-f3e7-4945-90a3-0533224b40d4 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:43.423754 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a52d1da-f3e7-4945-90a3-0533224b40d4 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:43.423754 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a52d1da-f3e7-4945-90a3-0533224b40d4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:43.445317 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-0a52d1da-f3e7-4945-90a3-0533224b40d4 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:47:43.446508 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-0a52d1da-f3e7-4945-90a3-0533224b40d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:47:43.511154 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1352/2679] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:47:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:47:43.519518 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bee8bad9-a607-4dcf-b79d-aaf6517c3352 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:43.519803 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bee8bad9-a607-4dcf-b79d-aaf6517c3352 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:43.520077 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bee8bad9-a607-4dcf-b79d-aaf6517c3352 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:43.542165 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-bee8bad9-a607-4dcf-b79d-aaf6517c3352 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:47:43.543561 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-bee8bad9-a607-4dcf-b79d-aaf6517c3352 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:47:43.551884 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1328/2680] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:47:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:47:43.556325 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a52d1da-f3e7-4945-90a3-0533224b40d4 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:43.566002 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a52d1da-f3e7-4945-90a3-0533224b40d4 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:43.570584 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc627113-3742-4915-ab0e-7556b32d86b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:43.571183 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc627113-3742-4915-ab0e-7556b32d86b2 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:43.571462 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc627113-3742-4915-ab0e-7556b32d86b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:43.571732 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc627113-3742-4915-ab0e-7556b32d86b2 admin admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:43.572374 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc627113-3742-4915-ab0e-7556b32d86b2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:47:43.572753 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc627113-3742-4915-ab0e-7556b32d86b2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:43.574354 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc627113-3742-4915-ab0e-7556b32d86b2 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:43.586219 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1353/2681] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Sep 25 04:47:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:47:43.590677 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bee8bad9-a607-4dcf-b79d-aaf6517c3352 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:43.600828 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bee8bad9-a607-4dcf-b79d-aaf6517c3352 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:43.605527 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f08d1ff9-d657-43b6-b137-614ce6faa483 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:43.606116 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f08d1ff9-d657-43b6-b137-614ce6faa483 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:43.606392 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f08d1ff9-d657-43b6-b137-614ce6faa483 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:43.606665 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f08d1ff9-d657-43b6-b137-614ce6faa483 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:43.607283 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f08d1ff9-d657-43b6-b137-614ce6faa483 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:43.609610 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f08d1ff9-d657-43b6-b137-614ce6faa483 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:43.629065 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=83,keystone:DELETE=14,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:47:43.641566 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1329/2682] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:47:43 2022] POST /identity/v3/projects => generated 373 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:47:43.642693 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc627113-3742-4915-ab0e-7556b32d86b2 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:43.652692 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc627113-3742-4915-ab0e-7556b32d86b2 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:43.671755 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:47:43.711494 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1688677f-2236-499c-91dc-c3bddd978ceb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:43.712107 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1688677f-2236-499c-91dc-c3bddd978ceb admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:43.712967 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1688677f-2236-499c-91dc-c3bddd978ceb admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:43.712967 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1688677f-2236-499c-91dc-c3bddd978ceb admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:43.713295 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1688677f-2236-499c-91dc-c3bddd978ceb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:43.715458 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1688677f-2236-499c-91dc-c3bddd978ceb admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:43.738883 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1354/2683] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:47:43 2022] POST /identity/v3/users => generated 287 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 04:47:43.743058 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f08d1ff9-d657-43b6-b137-614ce6faa483 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:43.753988 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f08d1ff9-d657-43b6-b137-614ce6faa483 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:43.758705 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7998924c-f676-4a90-970f-39cbce22d0de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:43.759317 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7998924c-f676-4a90-970f-39cbce22d0de admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:43.759714 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7998924c-f676-4a90-970f-39cbce22d0de admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:43.760018 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7998924c-f676-4a90-970f-39cbce22d0de admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:43.760686 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7998924c-f676-4a90-970f-39cbce22d0de admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:43.762548 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7998924c-f676-4a90-970f-39cbce22d0de admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:43.773776 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1330/2684] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:47:43 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:47:43.777691 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1688677f-2236-499c-91dc-c3bddd978ceb admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:43.788501 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1688677f-2236-499c-91dc-c3bddd978ceb admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:43.793341 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52c76c56-c91c-41b0-8b02-5514f9af9dac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:43.793949 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52c76c56-c91c-41b0-8b02-5514f9af9dac admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:43.794245 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52c76c56-c91c-41b0-8b02-5514f9af9dac admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:43.794509 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52c76c56-c91c-41b0-8b02-5514f9af9dac admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:43.795149 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52c76c56-c91c-41b0-8b02-5514f9af9dac admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:43.796928 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52c76c56-c91c-41b0-8b02-5514f9af9dac admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:43.807810 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1355/2685] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:47:43 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:47:43.812192 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7998924c-f676-4a90-970f-39cbce22d0de admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:43.822340 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7998924c-f676-4a90-970f-39cbce22d0de admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:43.827710 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a4d8a4e-855b-4c67-ae5a-3f95ca30d215 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:43.828588 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a4d8a4e-855b-4c67-ae5a-3f95ca30d215 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:43.829015 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a4d8a4e-855b-4c67-ae5a-3f95ca30d215 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:43.829411 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a4d8a4e-855b-4c67-ae5a-3f95ca30d215 admin admin] PATH_INFO: `/v3/projects/e546577d458b44ef801876e47407a8b1/users/3c04eac386504f9f8adaf85f39454365/roles/8f6cc574056c4278bc9778f14dc4b411` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:43.848865 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a4d8a4e-855b-4c67-ae5a-3f95ca30d215 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e546577d458b44ef801876e47407a8b1, user_id=3c04eac386504f9f8adaf85f39454365, role_id=8f6cc574056c4278bc9778f14dc4b411)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:43.851567 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a4d8a4e-855b-4c67-ae5a-3f95ca30d215 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:43.867039 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1331/2686] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:47:43 2022] PUT /identity/v3/projects/e546577d458b44ef801876e47407a8b1/users/3c04eac386504f9f8adaf85f39454365/roles/8f6cc574056c4278bc9778f14dc4b411 => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:47:43.872463 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ff8e712-e39f-43b3-beec-6cf7da5bad7c None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:43.872908 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ff8e712-e39f-43b3-beec-6cf7da5bad7c None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:43.877006 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ff8e712-e39f-43b3-beec-6cf7da5bad7c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:43.899719 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-3ff8e712-e39f-43b3-beec-6cf7da5bad7c None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:47:43.901106 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-3ff8e712-e39f-43b3-beec-6cf7da5bad7c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:47:43.951333 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a4d8a4e-855b-4c67-ae5a-3f95ca30d215 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:43.953952 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-7a4d8a4e-855b-4c67-ae5a-3f95ca30d215 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:47:43.976296 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1356/2687] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:47:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:47:43.986855 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ff8e712-e39f-43b3-beec-6cf7da5bad7c None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:43.988344 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-3ff8e712-e39f-43b3-beec-6cf7da5bad7c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:47:44.011367 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a4d8a4e-855b-4c67-ae5a-3f95ca30d215 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:44.015727 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b83d2d45-df14-4229-b378-42393b680c57 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:44.016272 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b83d2d45-df14-4229-b378-42393b680c57 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:44.016557 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b83d2d45-df14-4229-b378-42393b680c57 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:44.016863 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b83d2d45-df14-4229-b378-42393b680c57 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:44.018736 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-b83d2d45-df14-4229-b378-42393b680c57 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:47:44.031100 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b83d2d45-df14-4229-b378-42393b680c57 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:44.032537 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b83d2d45-df14-4229-b378-42393b680c57 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:44.043277 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1332/2688] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:47:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 25 04:47:44.051269 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ff8e712-e39f-43b3-beec-6cf7da5bad7c None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:44.055330 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-806e02f5-3b71-4e7e-9583-0faf30ad3a97 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:44.056086 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-806e02f5-3b71-4e7e-9583-0faf30ad3a97 service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:44.056383 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-806e02f5-3b71-4e7e-9583-0faf30ad3a97 service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:44.056708 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-806e02f5-3b71-4e7e-9583-0faf30ad3a97 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:44.070794 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-806e02f5-3b71-4e7e-9583-0faf30ad3a97 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:44.072348 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-806e02f5-3b71-4e7e-9583-0faf30ad3a97 service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:44.083549 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1357/2689] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:47:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 25 04:47:44.815879 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-923315e1-c476-4d01-be77-914e0eadffdf None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:44.816974 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-923315e1-c476-4d01-be77-914e0eadffdf None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:44.817254 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-923315e1-c476-4d01-be77-914e0eadffdf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:44.855256 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-923315e1-c476-4d01-be77-914e0eadffdf None None] MFA Rules not processed for user `3c04eac386504f9f8adaf85f39454365`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:47:44.856887 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-923315e1-c476-4d01-be77-914e0eadffdf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:47:44.929316 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1333/2690] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:47:44 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:47:44.934208 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-806e02f5-3b71-4e7e-9583-0faf30ad3a97 service neutron] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:44.944713 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-806e02f5-3b71-4e7e-9583-0faf30ad3a97 service neutron] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:44.951602 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83e4a75c-1b21-44af-b32c-e66222175989 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:44.953016 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e4a75c-1b21-44af-b32c-e66222175989 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:44.953369 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e4a75c-1b21-44af-b32c-e66222175989 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:44.953699 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e4a75c-1b21-44af-b32c-e66222175989 admin admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:44.954411 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83e4a75c-1b21-44af-b32c-e66222175989 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:44.956548 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83e4a75c-1b21-44af-b32c-e66222175989 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:44.996489 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1358/2691] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:47:44 2022] POST /identity/v3/projects => generated 373 bytes in 63 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:47:45.002169 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-923315e1-c476-4d01-be77-914e0eadffdf None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:45.015440 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-923315e1-c476-4d01-be77-914e0eadffdf None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:45.080229 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:47:45.095003 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04417586-db1a-4c63-bfb3-eaf341af3095 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:45.097461 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04417586-db1a-4c63-bfb3-eaf341af3095 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:45.098405 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04417586-db1a-4c63-bfb3-eaf341af3095 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:45.098691 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04417586-db1a-4c63-bfb3-eaf341af3095 admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:45.099370 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04417586-db1a-4c63-bfb3-eaf341af3095 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:45.101423 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04417586-db1a-4c63-bfb3-eaf341af3095 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:45.126987 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1334/2692] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:47:45 2022] POST /identity/v3/users => generated 293 bytes in 125 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:47:45.131185 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83e4a75c-1b21-44af-b32c-e66222175989 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:45.144402 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83e4a75c-1b21-44af-b32c-e66222175989 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:45.151119 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30ad202b-f5e1-441a-acc0-119d5a2b6538 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:45.151760 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30ad202b-f5e1-441a-acc0-119d5a2b6538 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:45.152042 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30ad202b-f5e1-441a-acc0-119d5a2b6538 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:45.152336 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30ad202b-f5e1-441a-acc0-119d5a2b6538 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:45.153031 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30ad202b-f5e1-441a-acc0-119d5a2b6538 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:45.154635 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30ad202b-f5e1-441a-acc0-119d5a2b6538 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:45.166122 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1359/2693] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:47:45 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:47:45.170354 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04417586-db1a-4c63-bfb3-eaf341af3095 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:45.183670 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04417586-db1a-4c63-bfb3-eaf341af3095 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:45.192110 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db1232d2-578c-468b-b5ca-2ffcc6087380 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:45.192905 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1232d2-578c-468b-b5ca-2ffcc6087380 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:45.193197 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1232d2-578c-468b-b5ca-2ffcc6087380 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:45.193505 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1232d2-578c-468b-b5ca-2ffcc6087380 admin admin] PATH_INFO: `/v3/projects/a3ad197b14d64415a70f9da43c71f34a/users/275d12c86072479e8d958a84d5c4c7e8/roles/c51fe7c5ad804040879acfb54269455f` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:45.212184 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db1232d2-578c-468b-b5ca-2ffcc6087380 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a3ad197b14d64415a70f9da43c71f34a, user_id=275d12c86072479e8d958a84d5c4c7e8, role_id=c51fe7c5ad804040879acfb54269455f)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:45.215179 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db1232d2-578c-468b-b5ca-2ffcc6087380 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:45.231859 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1335/2694] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:47:45 2022] PUT /identity/v3/projects/a3ad197b14d64415a70f9da43c71f34a/users/275d12c86072479e8d958a84d5c4c7e8/roles/c51fe7c5ad804040879acfb54269455f => generated 0 bytes in 62 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:47:45.236959 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30ad202b-f5e1-441a-acc0-119d5a2b6538 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:45.253862 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30ad202b-f5e1-441a-acc0-119d5a2b6538 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:45.266155 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:47:45.320178 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0ba7c5a-71e2-404d-bb35-9d861903bb40 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:45.320820 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0ba7c5a-71e2-404d-bb35-9d861903bb40 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:45.321100 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0ba7c5a-71e2-404d-bb35-9d861903bb40 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:45.321376 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0ba7c5a-71e2-404d-bb35-9d861903bb40 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:45.322009 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0ba7c5a-71e2-404d-bb35-9d861903bb40 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:45.323614 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0ba7c5a-71e2-404d-bb35-9d861903bb40 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:45.337253 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1360/2695] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:47:45 2022] GET /identity/v3/roles => generated 1420 bytes in 97 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:47:45.338184 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db1232d2-578c-468b-b5ca-2ffcc6087380 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:45.349281 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db1232d2-578c-468b-b5ca-2ffcc6087380 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:45.354236 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b23bcd2-a29c-4d01-90a1-6ece4e3cccca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:45.354634 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b23bcd2-a29c-4d01-90a1-6ece4e3cccca admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:45.354915 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b23bcd2-a29c-4d01-90a1-6ece4e3cccca admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:45.355196 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b23bcd2-a29c-4d01-90a1-6ece4e3cccca admin admin] PATH_INFO: `/v3/projects/a3ad197b14d64415a70f9da43c71f34a/users/275d12c86072479e8d958a84d5c4c7e8/roles/c51fe7c5ad804040879acfb54269455f` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:45.357149 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b23bcd2-a29c-4d01-90a1-6ece4e3cccca admin admin] RBAC: Authorizing `identity:create_grant(project_id=a3ad197b14d64415a70f9da43c71f34a, user_id=275d12c86072479e8d958a84d5c4c7e8, role_id=c51fe7c5ad804040879acfb54269455f)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:45.360340 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b23bcd2-a29c-4d01-90a1-6ece4e3cccca admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:45.372114 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1336/2696] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:47:45 2022] PUT /identity/v3/projects/a3ad197b14d64415a70f9da43c71f34a/users/275d12c86072479e8d958a84d5c4c7e8/roles/c51fe7c5ad804040879acfb54269455f => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:47:46.146520 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c903f02d-d722-470b-8990-d82d1f141dcf None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:46.147676 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c903f02d-d722-470b-8990-d82d1f141dcf None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:46.148298 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c903f02d-d722-470b-8990-d82d1f141dcf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:46.194456 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-c903f02d-d722-470b-8990-d82d1f141dcf None None] MFA Rules not processed for user `275d12c86072479e8d958a84d5c4c7e8`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:47:46.196011 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-c903f02d-d722-470b-8990-d82d1f141dcf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:47:46.298345 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1361/2697] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:47:46 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:47:46.317739 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b23bcd2-a29c-4d01-90a1-6ece4e3cccca admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:46.320607 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-1b23bcd2-a29c-4d01-90a1-6ece4e3cccca admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:47:46.390788 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b23bcd2-a29c-4d01-90a1-6ece4e3cccca admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:46.394547 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ce59e91-908f-459c-b709-f523445d7c2a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ffbc136c77b84e8c87128f7247f4ec03', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:46.395236 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce59e91-908f-459c-b709-f523445d7c2a service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:46.395590 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce59e91-908f-459c-b709-f523445d7c2a service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:46.395943 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce59e91-908f-459c-b709-f523445d7c2a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:46.481710 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ce59e91-908f-459c-b709-f523445d7c2a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:46.483953 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ce59e91-908f-459c-b709-f523445d7c2a service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:46.493688 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1337/2698] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:47:46 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 177 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:47:47.035126 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c903f02d-d722-470b-8990-d82d1f141dcf None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:47.036744 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-c903f02d-d722-470b-8990-d82d1f141dcf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:47:47.104164 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c903f02d-d722-470b-8990-d82d1f141dcf None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:47.108598 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc700535-5afc-401b-ba91-3279f66397c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:47.109254 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc700535-5afc-401b-ba91-3279f66397c7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:47.109530 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc700535-5afc-401b-ba91-3279f66397c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:47.109818 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc700535-5afc-401b-ba91-3279f66397c7 admin admin] PATH_INFO: `/v3/users/30ad4e204c60461ab9cc3ba16b377a21` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:47.110924 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc700535-5afc-401b-ba91-3279f66397c7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=30ad4e204c60461ab9cc3ba16b377a21)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:47.112862 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc700535-5afc-401b-ba91-3279f66397c7 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:47.155341 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:47:47.165885 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-dc700535-5afc-401b-ba91-3279f66397c7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '30ad4e204c60461ab9cc3ba16b377a21'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:47.176132 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-dc700535-5afc-401b-ba91-3279f66397c7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '30ad4e204c60461ab9cc3ba16b377a21'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:47.190185 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-dc700535-5afc-401b-ba91-3279f66397c7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '30ad4e204c60461ab9cc3ba16b377a21'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:47.190859 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-dc700535-5afc-401b-ba91-3279f66397c7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '30ad4e204c60461ab9cc3ba16b377a21'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:47.197980 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-dc700535-5afc-401b-ba91-3279f66397c7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '30ad4e204c60461ab9cc3ba16b377a21'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:47.205377 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1362/2699] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:47:47 2022] DELETE /identity/v3/users/30ad4e204c60461ab9cc3ba16b377a21 => generated 0 bytes in 171 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:47:47.209968 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de94a36f-db8b-4e06-be94-bebeab9aa339 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:47.210256 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de94a36f-db8b-4e06-be94-bebeab9aa339 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:47.210528 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de94a36f-db8b-4e06-be94-bebeab9aa339 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:47.234659 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-de94a36f-db8b-4e06-be94-bebeab9aa339 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:47:47.236058 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-de94a36f-db8b-4e06-be94-bebeab9aa339 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:47:47.343300 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1338/2700] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:47:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:47:47.353405 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc700535-5afc-401b-ba91-3279f66397c7 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:47.355117 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-dc700535-5afc-401b-ba91-3279f66397c7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:47:47.448465 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc700535-5afc-401b-ba91-3279f66397c7 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:47.452959 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24fab80f-4c4d-4bfe-8e3a-518fe6a485ac service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:47.453577 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24fab80f-4c4d-4bfe-8e3a-518fe6a485ac service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:47.453857 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24fab80f-4c4d-4bfe-8e3a-518fe6a485ac service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:47.454150 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24fab80f-4c4d-4bfe-8e3a-518fe6a485ac service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:47.463986 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24fab80f-4c4d-4bfe-8e3a-518fe6a485ac service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:47.465739 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24fab80f-4c4d-4bfe-8e3a-518fe6a485ac service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:47.488741 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1363/2701] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:47:47 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 131 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:47:47.767744 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de94a36f-db8b-4e06-be94-bebeab9aa339 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:47.769604 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-de94a36f-db8b-4e06-be94-bebeab9aa339 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:47:47.778540 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:47:47.786261 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de94a36f-db8b-4e06-be94-bebeab9aa339 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:47.791783 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-672c3688-c88b-406f-a225-9e186fe8571b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:47.792993 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-672c3688-c88b-406f-a225-9e186fe8571b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:47.794103 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-672c3688-c88b-406f-a225-9e186fe8571b admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:47.795000 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-672c3688-c88b-406f-a225-9e186fe8571b admin admin] PATH_INFO: `/v3/projects/ae42d8eb861e43b4a514d1cb5ec7ec5b` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:47.798410 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-672c3688-c88b-406f-a225-9e186fe8571b admin admin] RBAC: Authorizing `identity:delete_project(project_id=ae42d8eb861e43b4a514d1cb5ec7ec5b)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:47.800625 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-672c3688-c88b-406f-a225-9e186fe8571b admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:47.859333 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-672c3688-c88b-406f-a225-9e186fe8571b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ae42d8eb861e43b4a514d1cb5ec7ec5b', 'request_id': 'req-672c3688-c88b-406f-a225-9e186fe8571b'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:47.869194 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-672c3688-c88b-406f-a225-9e186fe8571b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ae42d8eb861e43b4a514d1cb5ec7ec5b', 'request_id': 'req-672c3688-c88b-406f-a225-9e186fe8571b'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:47.888688 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-672c3688-c88b-406f-a225-9e186fe8571b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ae42d8eb861e43b4a514d1cb5ec7ec5b', 'request_id': 'req-672c3688-c88b-406f-a225-9e186fe8571b'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:47.889362 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-672c3688-c88b-406f-a225-9e186fe8571b admin admin] The token cache is being invalidate because project ae42d8eb861e43b4a514d1cb5ec7ec5b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:47:47.889739 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-672c3688-c88b-406f-a225-9e186fe8571b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:47.890437 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-672c3688-c88b-406f-a225-9e186fe8571b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:47.892137 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1339/2702] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:47:47 2022] DELETE /identity/v3/projects/ae42d8eb861e43b4a514d1cb5ec7ec5b => generated 0 bytes in 125 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:47:47.896801 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e10f5a1-984e-466a-9652-26ef5c681757 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:47.897095 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e10f5a1-984e-466a-9652-26ef5c681757 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:47.897371 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e10f5a1-984e-466a-9652-26ef5c681757 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:47.897574 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-672c3688-c88b-406f-a225-9e186fe8571b admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:47.899694 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-672c3688-c88b-406f-a225-9e186fe8571b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:47:47.922206 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-0e10f5a1-984e-466a-9652-26ef5c681757 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:47:47.923694 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-0e10f5a1-984e-466a-9652-26ef5c681757 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:47:47.971731 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-672c3688-c88b-406f-a225-9e186fe8571b admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:47.976832 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-713a67cd-4c1f-40d9-9a4b-b3f02a0fc6de service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ffbc136c77b84e8c87128f7247f4ec03', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:47.977482 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-713a67cd-4c1f-40d9-9a4b-b3f02a0fc6de service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:47.977735 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-713a67cd-4c1f-40d9-9a4b-b3f02a0fc6de service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:47.978006 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-713a67cd-4c1f-40d9-9a4b-b3f02a0fc6de service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:47.979999 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-713a67cd-4c1f-40d9-9a4b-b3f02a0fc6de service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:47:47.994538 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1364/2703] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:47:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:47:48.002699 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf8d2a3b-edb2-4fa6-9e48-9876e3ac22c8 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:48.002982 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf8d2a3b-edb2-4fa6-9e48-9876e3ac22c8 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:48.003264 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf8d2a3b-edb2-4fa6-9e48-9876e3ac22c8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:48.026502 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-bf8d2a3b-edb2-4fa6-9e48-9876e3ac22c8 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:47:48.027735 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-bf8d2a3b-edb2-4fa6-9e48-9876e3ac22c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:47:48.040782 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1365/2704] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:47:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:47:48.041588 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf8d2a3b-edb2-4fa6-9e48-9876e3ac22c8 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:48.052168 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf8d2a3b-edb2-4fa6-9e48-9876e3ac22c8 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:48.055311 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-713a67cd-4c1f-40d9-9a4b-b3f02a0fc6de service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:48.056791 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-713a67cd-4c1f-40d9-9a4b-b3f02a0fc6de service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:48.057376 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80a4c4e8-82af-44cc-b457-d6c53cdce911 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:48.058056 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a4c4e8-82af-44cc-b457-d6c53cdce911 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:48.058404 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a4c4e8-82af-44cc-b457-d6c53cdce911 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:48.058743 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a4c4e8-82af-44cc-b457-d6c53cdce911 admin admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:48.059457 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80a4c4e8-82af-44cc-b457-d6c53cdce911 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:47:48.059825 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80a4c4e8-82af-44cc-b457-d6c53cdce911 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:48.062128 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80a4c4e8-82af-44cc-b457-d6c53cdce911 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:48.069220 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1340/2705] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:47:47 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 172 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:47:48.076834 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1366/2706] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Sep 25 04:47:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:47:48.081051 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-713a67cd-4c1f-40d9-9a4b-b3f02a0fc6de service cinder] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:48.094371 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-713a67cd-4c1f-40d9-9a4b-b3f02a0fc6de service cinder] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:48.101868 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84fc15e7-71fe-4ae2-8c5e-3888dad888d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:48.102961 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84fc15e7-71fe-4ae2-8c5e-3888dad888d5 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:48.102961 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84fc15e7-71fe-4ae2-8c5e-3888dad888d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:48.103172 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84fc15e7-71fe-4ae2-8c5e-3888dad888d5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:48.103648 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84fc15e7-71fe-4ae2-8c5e-3888dad888d5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:48.106288 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84fc15e7-71fe-4ae2-8c5e-3888dad888d5 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:48.137344 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1341/2707] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:47:48 2022] POST /identity/v3/projects => generated 391 bytes in 57 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 04:47:48.142246 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80a4c4e8-82af-44cc-b457-d6c53cdce911 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:48.154421 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80a4c4e8-82af-44cc-b457-d6c53cdce911 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:48.195538 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:47:48.229292 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bf915c1-4875-48e2-a771-b76bfba27458 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:48.230042 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bf915c1-4875-48e2-a771-b76bfba27458 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:48.230420 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bf915c1-4875-48e2-a771-b76bfba27458 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:48.230763 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bf915c1-4875-48e2-a771-b76bfba27458 admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:48.231466 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bf915c1-4875-48e2-a771-b76bfba27458 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:48.234648 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bf915c1-4875-48e2-a771-b76bfba27458 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:48.261107 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1367/2708] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:47:48 2022] POST /identity/v3/users => generated 296 bytes in 119 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:47:48.265602 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84fc15e7-71fe-4ae2-8c5e-3888dad888d5 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:48.280987 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84fc15e7-71fe-4ae2-8c5e-3888dad888d5 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:48.287771 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c18491a5-a580-46eb-a3eb-dc31c11fed0f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:48.288419 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c18491a5-a580-46eb-a3eb-dc31c11fed0f admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:48.288733 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c18491a5-a580-46eb-a3eb-dc31c11fed0f admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:48.289020 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c18491a5-a580-46eb-a3eb-dc31c11fed0f admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:48.289672 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c18491a5-a580-46eb-a3eb-dc31c11fed0f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:48.291353 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c18491a5-a580-46eb-a3eb-dc31c11fed0f admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:48.307506 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1342/2709] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:47:48 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:47:48.312739 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bf915c1-4875-48e2-a771-b76bfba27458 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:48.324832 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bf915c1-4875-48e2-a771-b76bfba27458 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:48.328249 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d47d738f-4c0e-471b-a7da-89b19cc76d36 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:48.328900 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d47d738f-4c0e-471b-a7da-89b19cc76d36 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:48.329186 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d47d738f-4c0e-471b-a7da-89b19cc76d36 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:48.329456 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d47d738f-4c0e-471b-a7da-89b19cc76d36 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:48.330273 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d47d738f-4c0e-471b-a7da-89b19cc76d36 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:48.331953 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d47d738f-4c0e-471b-a7da-89b19cc76d36 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:48.346328 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1368/2710] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:47:48 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:47:48.350851 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c18491a5-a580-46eb-a3eb-dc31c11fed0f admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:48.356582 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:47:48.363541 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c18491a5-a580-46eb-a3eb-dc31c11fed0f admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:48.369715 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42001afe-9c54-4965-a4da-954c7bb9a7c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:48.370411 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42001afe-9c54-4965-a4da-954c7bb9a7c1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:48.370696 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42001afe-9c54-4965-a4da-954c7bb9a7c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:48.370976 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42001afe-9c54-4965-a4da-954c7bb9a7c1 admin admin] PATH_INFO: `/v3/projects/17e4e4f8d95b4999ae6a942a754d2916/users/329760399a2b473faee554c2c29c117f/roles/8f6cc574056c4278bc9778f14dc4b411` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:48.394032 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42001afe-9c54-4965-a4da-954c7bb9a7c1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=17e4e4f8d95b4999ae6a942a754d2916, user_id=329760399a2b473faee554c2c29c117f, role_id=8f6cc574056c4278bc9778f14dc4b411)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:48.397012 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42001afe-9c54-4965-a4da-954c7bb9a7c1 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:48.412782 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1343/2711] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:47:48 2022] PUT /identity/v3/projects/17e4e4f8d95b4999ae6a942a754d2916/users/329760399a2b473faee554c2c29c117f/roles/8f6cc574056c4278bc9778f14dc4b411 => generated 0 bytes in 63 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:47:48.423763 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4b0e4e3-e043-4640-a55d-93ea2564acca None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:48.424101 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4b0e4e3-e043-4640-a55d-93ea2564acca None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:48.424317 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4b0e4e3-e043-4640-a55d-93ea2564acca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:48.458215 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-f4b0e4e3-e043-4640-a55d-93ea2564acca None None] MFA Rules not processed for user `329760399a2b473faee554c2c29c117f`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:47:48.459798 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-f4b0e4e3-e043-4640-a55d-93ea2564acca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:47:48.537448 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1369/2712] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:47:48 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:47:48.549101 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42001afe-9c54-4965-a4da-954c7bb9a7c1 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:48.619625 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42001afe-9c54-4965-a4da-954c7bb9a7c1 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:48.623300 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0071372-69aa-4e2d-b1eb-8a599a96c778 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ffbc136c77b84e8c87128f7247f4ec03', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:48.624609 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0071372-69aa-4e2d-b1eb-8a599a96c778 service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:48.624609 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0071372-69aa-4e2d-b1eb-8a599a96c778 service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:48.624941 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0071372-69aa-4e2d-b1eb-8a599a96c778 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:48.633796 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0071372-69aa-4e2d-b1eb-8a599a96c778 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:48.635287 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0071372-69aa-4e2d-b1eb-8a599a96c778 service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:48.644400 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1344/2713] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:47:48 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 25 04:47:51.188800 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4b0e4e3-e043-4640-a55d-93ea2564acca None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:51.190355 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-f4b0e4e3-e043-4640-a55d-93ea2564acca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:47:51.260889 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4b0e4e3-e043-4640-a55d-93ea2564acca None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:51.266205 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31c6ae6b-0d0f-4794-a63d-6b15cc42bc29 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:51.266866 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31c6ae6b-0d0f-4794-a63d-6b15cc42bc29 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:51.267145 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31c6ae6b-0d0f-4794-a63d-6b15cc42bc29 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:51.267469 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31c6ae6b-0d0f-4794-a63d-6b15cc42bc29 admin admin] PATH_INFO: `/v3/users/329e3f9810b648218dd45a8b489adb1d` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:51.283798 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31c6ae6b-0d0f-4794-a63d-6b15cc42bc29 admin admin] RBAC: Authorizing `identity:delete_user(user_id=329e3f9810b648218dd45a8b489adb1d)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:51.285775 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31c6ae6b-0d0f-4794-a63d-6b15cc42bc29 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:51.320785 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=92,keystone:INSERT=3,keystone:DELETE=5 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:47:51.331882 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-31c6ae6b-0d0f-4794-a63d-6b15cc42bc29 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '329e3f9810b648218dd45a8b489adb1d'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:51.338480 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-31c6ae6b-0d0f-4794-a63d-6b15cc42bc29 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '329e3f9810b648218dd45a8b489adb1d'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:51.352461 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-31c6ae6b-0d0f-4794-a63d-6b15cc42bc29 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '329e3f9810b648218dd45a8b489adb1d'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:51.353154 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-31c6ae6b-0d0f-4794-a63d-6b15cc42bc29 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '329e3f9810b648218dd45a8b489adb1d'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:51.359941 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-31c6ae6b-0d0f-4794-a63d-6b15cc42bc29 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '329e3f9810b648218dd45a8b489adb1d'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:51.366054 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1370/2714] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:47:51 2022] DELETE /identity/v3/users/329e3f9810b648218dd45a8b489adb1d => generated 0 bytes in 178 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 04:47:51.537994 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0071372-69aa-4e2d-b1eb-8a599a96c778 service cinder] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:51.549470 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0071372-69aa-4e2d-b1eb-8a599a96c778 service cinder] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:51.607911 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f1e7e34-5d92-4444-b66b-9276f74266aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:51.608573 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f1e7e34-5d92-4444-b66b-9276f74266aa admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:51.608888 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f1e7e34-5d92-4444-b66b-9276f74266aa admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:51.609166 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f1e7e34-5d92-4444-b66b-9276f74266aa admin admin] PATH_INFO: `/v3/projects/00f66d2c942b419abc8d9d010c5a57f3` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:51.619300 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f1e7e34-5d92-4444-b66b-9276f74266aa admin admin] RBAC: Authorizing `identity:delete_project(project_id=00f66d2c942b419abc8d9d010c5a57f3)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:51.621248 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f1e7e34-5d92-4444-b66b-9276f74266aa admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:51.666691 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-6f1e7e34-5d92-4444-b66b-9276f74266aa admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '00f66d2c942b419abc8d9d010c5a57f3', 'request_id': 'req-6f1e7e34-5d92-4444-b66b-9276f74266aa'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:51.673579 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-6f1e7e34-5d92-4444-b66b-9276f74266aa admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '00f66d2c942b419abc8d9d010c5a57f3', 'request_id': 'req-6f1e7e34-5d92-4444-b66b-9276f74266aa'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:51.679222 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-6f1e7e34-5d92-4444-b66b-9276f74266aa admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '00f66d2c942b419abc8d9d010c5a57f3', 'request_id': 'req-6f1e7e34-5d92-4444-b66b-9276f74266aa'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:51.691352 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-6f1e7e34-5d92-4444-b66b-9276f74266aa admin admin] The token cache is being invalidate because project 00f66d2c942b419abc8d9d010c5a57f3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:47:51.691667 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-6f1e7e34-5d92-4444-b66b-9276f74266aa admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:51.692311 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-6f1e7e34-5d92-4444-b66b-9276f74266aa admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:51.693822 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1345/2715] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:47:51 2022] DELETE /identity/v3/projects/00f66d2c942b419abc8d9d010c5a57f3 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:47:51.698021 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31c6ae6b-0d0f-4794-a63d-6b15cc42bc29 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:51.699548 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-31c6ae6b-0d0f-4794-a63d-6b15cc42bc29 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:47:51.713876 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31c6ae6b-0d0f-4794-a63d-6b15cc42bc29 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:51.769043 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90457d70-1091-4dae-bead-0c52a5e38164 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:51.769675 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90457d70-1091-4dae-bead-0c52a5e38164 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:51.769953 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90457d70-1091-4dae-bead-0c52a5e38164 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:51.770225 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90457d70-1091-4dae-bead-0c52a5e38164 admin admin] PATH_INFO: `/v3/users/6036528f63e3402283c4f3f96adf5f6e` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:51.783272 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90457d70-1091-4dae-bead-0c52a5e38164 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6036528f63e3402283c4f3f96adf5f6e)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:51.785161 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90457d70-1091-4dae-bead-0c52a5e38164 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:51.832468 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-90457d70-1091-4dae-bead-0c52a5e38164 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6036528f63e3402283c4f3f96adf5f6e'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:51.840477 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-90457d70-1091-4dae-bead-0c52a5e38164 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6036528f63e3402283c4f3f96adf5f6e'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:51.852105 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-90457d70-1091-4dae-bead-0c52a5e38164 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6036528f63e3402283c4f3f96adf5f6e'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:51.852844 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-90457d70-1091-4dae-bead-0c52a5e38164 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6036528f63e3402283c4f3f96adf5f6e'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:51.860123 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-90457d70-1091-4dae-bead-0c52a5e38164 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6036528f63e3402283c4f3f96adf5f6e'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:51.866055 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1371/2716] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:47:51 2022] DELETE /identity/v3/users/6036528f63e3402283c4f3f96adf5f6e => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:47:52.034151 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f1e7e34-5d92-4444-b66b-9276f74266aa admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:47:52.035523 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-6f1e7e34-5d92-4444-b66b-9276f74266aa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:47:52.045060 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:47:52.050032 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f1e7e34-5d92-4444-b66b-9276f74266aa admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:47:52.107621 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-710ac6fa-57f3-4ed5-8196-b32279551c56 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:47:52.109391 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-710ac6fa-57f3-4ed5-8196-b32279551c56 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:47:52.109668 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-710ac6fa-57f3-4ed5-8196-b32279551c56 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:47:52.109942 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-710ac6fa-57f3-4ed5-8196-b32279551c56 admin admin] PATH_INFO: `/v3/projects/b06e471023194d10920aade4346c3bc3` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:47:52.120828 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-710ac6fa-57f3-4ed5-8196-b32279551c56 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b06e471023194d10920aade4346c3bc3)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:47:52.122752 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-710ac6fa-57f3-4ed5-8196-b32279551c56 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:47:52.169805 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-710ac6fa-57f3-4ed5-8196-b32279551c56 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b06e471023194d10920aade4346c3bc3', 'request_id': 'req-710ac6fa-57f3-4ed5-8196-b32279551c56'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:52.177594 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-710ac6fa-57f3-4ed5-8196-b32279551c56 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b06e471023194d10920aade4346c3bc3', 'request_id': 'req-710ac6fa-57f3-4ed5-8196-b32279551c56'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:52.183144 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-710ac6fa-57f3-4ed5-8196-b32279551c56 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b06e471023194d10920aade4346c3bc3', 'request_id': 'req-710ac6fa-57f3-4ed5-8196-b32279551c56'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:52.195774 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-710ac6fa-57f3-4ed5-8196-b32279551c56 admin admin] The token cache is being invalidate because project b06e471023194d10920aade4346c3bc3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:47:52.196069 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-710ac6fa-57f3-4ed5-8196-b32279551c56 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:52.196737 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-710ac6fa-57f3-4ed5-8196-b32279551c56 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:47:52.198202 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1346/2717] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:47:52 2022] DELETE /identity/v3/projects/b06e471023194d10920aade4346c3bc3 => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:48:01.862742 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=74,keystone:DELETE=15,keystone:INSERT=2 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:48:02.192707 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=54,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:48:11.116587 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90457d70-1091-4dae-bead-0c52a5e38164 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:11.118216 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-90457d70-1091-4dae-bead-0c52a5e38164 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:48:11.131678 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90457d70-1091-4dae-bead-0c52a5e38164 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:11.188117 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f259ea8c-a43f-4846-be4b-30602dce8c4f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:11.188773 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f259ea8c-a43f-4846-be4b-30602dce8c4f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:11.189056 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f259ea8c-a43f-4846-be4b-30602dce8c4f admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:11.189328 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f259ea8c-a43f-4846-be4b-30602dce8c4f admin admin] PATH_INFO: `/v3/users/329760399a2b473faee554c2c29c117f` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:11.190451 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f259ea8c-a43f-4846-be4b-30602dce8c4f admin admin] RBAC: Authorizing `identity:delete_user(user_id=329760399a2b473faee554c2c29c117f)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:11.192350 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f259ea8c-a43f-4846-be4b-30602dce8c4f admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:11.237331 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-f259ea8c-a43f-4846-be4b-30602dce8c4f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '329760399a2b473faee554c2c29c117f'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:11.244661 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-f259ea8c-a43f-4846-be4b-30602dce8c4f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '329760399a2b473faee554c2c29c117f'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:11.257199 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-f259ea8c-a43f-4846-be4b-30602dce8c4f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '329760399a2b473faee554c2c29c117f'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:11.257843 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-f259ea8c-a43f-4846-be4b-30602dce8c4f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '329760399a2b473faee554c2c29c117f'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:11.265536 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-f259ea8c-a43f-4846-be4b-30602dce8c4f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '329760399a2b473faee554c2c29c117f'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:11.271428 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1372/2718] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:48:11 2022] DELETE /identity/v3/users/329760399a2b473faee554c2c29c117f => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:48:11.275882 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-398926bb-17b7-4ab3-9af0-6a940456d8d0 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:11.276359 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-398926bb-17b7-4ab3-9af0-6a940456d8d0 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:11.276484 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-398926bb-17b7-4ab3-9af0-6a940456d8d0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:11.296627 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-398926bb-17b7-4ab3-9af0-6a940456d8d0 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:48:11.297872 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-398926bb-17b7-4ab3-9af0-6a940456d8d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:48:11.355818 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1347/2719] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:48:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:48:11.366235 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f259ea8c-a43f-4846-be4b-30602dce8c4f admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:11.367637 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-f259ea8c-a43f-4846-be4b-30602dce8c4f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:48:11.423855 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f259ea8c-a43f-4846-be4b-30602dce8c4f admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:11.427516 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3534d540-bb13-42d2-8d64-0f7f05955ea1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:11.428120 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3534d540-bb13-42d2-8d64-0f7f05955ea1 service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:11.428404 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3534d540-bb13-42d2-8d64-0f7f05955ea1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:11.428702 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3534d540-bb13-42d2-8d64-0f7f05955ea1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:11.438147 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3534d540-bb13-42d2-8d64-0f7f05955ea1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:11.439573 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3534d540-bb13-42d2-8d64-0f7f05955ea1 service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:11.448956 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1373/2720] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:48:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:48:11.711973 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-398926bb-17b7-4ab3-9af0-6a940456d8d0 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:11.713562 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-398926bb-17b7-4ab3-9af0-6a940456d8d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:48:11.728016 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-398926bb-17b7-4ab3-9af0-6a940456d8d0 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:11.734063 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57f7220a-781a-46d8-a9d2-9287a1aadcb8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:11.734784 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57f7220a-781a-46d8-a9d2-9287a1aadcb8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:11.735149 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57f7220a-781a-46d8-a9d2-9287a1aadcb8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:11.735327 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57f7220a-781a-46d8-a9d2-9287a1aadcb8 admin admin] PATH_INFO: `/v3/projects/17e4e4f8d95b4999ae6a942a754d2916` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:11.736442 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57f7220a-781a-46d8-a9d2-9287a1aadcb8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=17e4e4f8d95b4999ae6a942a754d2916)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:11.739282 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57f7220a-781a-46d8-a9d2-9287a1aadcb8 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:11.786085 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-57f7220a-781a-46d8-a9d2-9287a1aadcb8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '17e4e4f8d95b4999ae6a942a754d2916', 'request_id': 'req-57f7220a-781a-46d8-a9d2-9287a1aadcb8'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:11.793048 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-57f7220a-781a-46d8-a9d2-9287a1aadcb8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '17e4e4f8d95b4999ae6a942a754d2916', 'request_id': 'req-57f7220a-781a-46d8-a9d2-9287a1aadcb8'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:11.798783 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-57f7220a-781a-46d8-a9d2-9287a1aadcb8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '17e4e4f8d95b4999ae6a942a754d2916', 'request_id': 'req-57f7220a-781a-46d8-a9d2-9287a1aadcb8'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:11.811764 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-57f7220a-781a-46d8-a9d2-9287a1aadcb8 admin admin] The token cache is being invalidate because project 17e4e4f8d95b4999ae6a942a754d2916 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:48:11.812264 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-57f7220a-781a-46d8-a9d2-9287a1aadcb8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:11.813054 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-57f7220a-781a-46d8-a9d2-9287a1aadcb8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:11.814569 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1348/2721] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:48:11 2022] DELETE /identity/v3/projects/17e4e4f8d95b4999ae6a942a754d2916 => generated 0 bytes in 103 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:48:11.819710 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14586dc9-d737-4483-9596-f479c03eecd8 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:11.820129 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14586dc9-d737-4483-9596-f479c03eecd8 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:11.820436 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14586dc9-d737-4483-9596-f479c03eecd8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:11.842399 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-14586dc9-d737-4483-9596-f479c03eecd8 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:48:11.843607 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-14586dc9-d737-4483-9596-f479c03eecd8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:48:11.860807 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:48:11.911817 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1374/2722] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:48:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:48:11.977172 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5d12250-00f6-4bfe-a382-dcb97954eff4 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:11.977459 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5d12250-00f6-4bfe-a382-dcb97954eff4 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:11.977739 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5d12250-00f6-4bfe-a382-dcb97954eff4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:11.997679 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-a5d12250-00f6-4bfe-a382-dcb97954eff4 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:48:11.999619 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-a5d12250-00f6-4bfe-a382-dcb97954eff4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:48:12.007622 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1349/2723] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:48:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:48:12.011813 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14586dc9-d737-4483-9596-f479c03eecd8 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:12.022682 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14586dc9-d737-4483-9596-f479c03eecd8 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:12.027281 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45f0286f-7072-4b01-ab56-5432e1b837c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:12.027913 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45f0286f-7072-4b01-ab56-5432e1b837c1 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:12.028188 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45f0286f-7072-4b01-ab56-5432e1b837c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:12.028455 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45f0286f-7072-4b01-ab56-5432e1b837c1 admin admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:12.029190 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45f0286f-7072-4b01-ab56-5432e1b837c1 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:48:12.029744 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45f0286f-7072-4b01-ab56-5432e1b837c1 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:12.031331 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45f0286f-7072-4b01-ab56-5432e1b837c1 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:12.042927 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1375/2724] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Sep 25 04:48:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:48:12.048034 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5d12250-00f6-4bfe-a382-dcb97954eff4 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:12.057724 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5d12250-00f6-4bfe-a382-dcb97954eff4 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:12.063599 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19c20db7-cf9e-4298-bb73-a6667d751626 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:12.064209 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c20db7-cf9e-4298-bb73-a6667d751626 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:12.064489 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c20db7-cf9e-4298-bb73-a6667d751626 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:12.064796 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c20db7-cf9e-4298-bb73-a6667d751626 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:12.065458 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19c20db7-cf9e-4298-bb73-a6667d751626 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:12.067372 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19c20db7-cf9e-4298-bb73-a6667d751626 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:12.097342 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1350/2725] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:48:12 2022] POST /identity/v3/projects => generated 399 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 04:48:12.101391 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45f0286f-7072-4b01-ab56-5432e1b837c1 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:12.113645 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45f0286f-7072-4b01-ab56-5432e1b837c1 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:12.178758 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-274de1e0-2302-402f-9463-80567505403b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:12.179462 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-274de1e0-2302-402f-9463-80567505403b admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:12.179819 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-274de1e0-2302-402f-9463-80567505403b admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:12.180098 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-274de1e0-2302-402f-9463-80567505403b admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:12.180763 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-274de1e0-2302-402f-9463-80567505403b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:12.182742 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-274de1e0-2302-402f-9463-80567505403b admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:12.204466 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1376/2726] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:48:12 2022] POST /identity/v3/users => generated 300 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:48:12.208585 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19c20db7-cf9e-4298-bb73-a6667d751626 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:12.219887 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19c20db7-cf9e-4298-bb73-a6667d751626 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:12.224731 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7846fe6f-df38-4825-adff-327f041f86c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:12.225344 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7846fe6f-df38-4825-adff-327f041f86c5 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:12.225624 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7846fe6f-df38-4825-adff-327f041f86c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:12.225926 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7846fe6f-df38-4825-adff-327f041f86c5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:12.226572 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7846fe6f-df38-4825-adff-327f041f86c5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:12.228190 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7846fe6f-df38-4825-adff-327f041f86c5 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:12.239109 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1351/2727] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:48:12 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:48:12.243302 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-274de1e0-2302-402f-9463-80567505403b admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:12.253836 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-274de1e0-2302-402f-9463-80567505403b admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:12.258589 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b246f835-1082-48e0-a0e4-618dc41e6d9c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:12.259198 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b246f835-1082-48e0-a0e4-618dc41e6d9c admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:12.259479 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b246f835-1082-48e0-a0e4-618dc41e6d9c admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:12.259750 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b246f835-1082-48e0-a0e4-618dc41e6d9c admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:12.260390 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b246f835-1082-48e0-a0e4-618dc41e6d9c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:12.262019 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b246f835-1082-48e0-a0e4-618dc41e6d9c admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:12.273384 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1377/2728] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:48:12 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 25 04:48:12.277552 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7846fe6f-df38-4825-adff-327f041f86c5 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:12.287623 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7846fe6f-df38-4825-adff-327f041f86c5 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:12.292450 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c92bd7fd-09d8-490d-8690-f055808cb354 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:12.293142 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c92bd7fd-09d8-490d-8690-f055808cb354 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:12.293423 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c92bd7fd-09d8-490d-8690-f055808cb354 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:12.293698 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c92bd7fd-09d8-490d-8690-f055808cb354 admin admin] PATH_INFO: `/v3/projects/df5dfe4898574c00ab5e51ccbb027be6/users/fb87a4a326b6420d8b93784ec0def9f7/roles/8f6cc574056c4278bc9778f14dc4b411` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:12.309570 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c92bd7fd-09d8-490d-8690-f055808cb354 admin admin] RBAC: Authorizing `identity:create_grant(project_id=df5dfe4898574c00ab5e51ccbb027be6, user_id=fb87a4a326b6420d8b93784ec0def9f7, role_id=8f6cc574056c4278bc9778f14dc4b411)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:12.312994 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c92bd7fd-09d8-490d-8690-f055808cb354 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:12.331395 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1352/2729] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:48:12 2022] PUT /identity/v3/projects/df5dfe4898574c00ab5e51ccbb027be6/users/fb87a4a326b6420d8b93784ec0def9f7/roles/8f6cc574056c4278bc9778f14dc4b411 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:48:12.341759 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d6f41f4-b803-487c-9675-cbcb7e9f4823 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:12.342245 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d6f41f4-b803-487c-9675-cbcb7e9f4823 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:12.342529 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d6f41f4-b803-487c-9675-cbcb7e9f4823 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:12.372098 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-6d6f41f4-b803-487c-9675-cbcb7e9f4823 None None] MFA Rules not processed for user `fb87a4a326b6420d8b93784ec0def9f7`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:48:12.373949 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-6d6f41f4-b803-487c-9675-cbcb7e9f4823 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:48:12.390245 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c92bd7fd-09d8-490d-8690-f055808cb354 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:12.392050 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-c92bd7fd-09d8-490d-8690-f055808cb354 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:48:12.415490 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:48:12.468717 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1378/2730] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:48:12 2022] POST /identity/v3/auth/tokens => generated 3276 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:48:12.476731 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d6f41f4-b803-487c-9675-cbcb7e9f4823 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:12.478913 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-6d6f41f4-b803-487c-9675-cbcb7e9f4823 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:48:12.483618 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c92bd7fd-09d8-490d-8690-f055808cb354 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:12.489996 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ef062b0-8817-44c1-93ed-4d5ab5d0e37a service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '676d04ef5a984edcaad63d9156568660', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:12.490812 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef062b0-8817-44c1-93ed-4d5ab5d0e37a service placement] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:12.491194 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef062b0-8817-44c1-93ed-4d5ab5d0e37a service placement] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:12.491579 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef062b0-8817-44c1-93ed-4d5ab5d0e37a service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:12.494112 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-4ef062b0-8817-44c1-93ed-4d5ab5d0e37a service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:48:12.503059 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:48:12.507562 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ef062b0-8817-44c1-93ed-4d5ab5d0e37a service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:12.509211 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ef062b0-8817-44c1-93ed-4d5ab5d0e37a service placement] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:12.522655 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1353/2731] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:48:12 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 133 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 25 04:48:12.548690 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d6f41f4-b803-487c-9675-cbcb7e9f4823 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:12.553386 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d1577c2-ab20-42d1-a25f-9c3b37739537 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ffbc136c77b84e8c87128f7247f4ec03', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:12.553741 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1577c2-ab20-42d1-a25f-9c3b37739537 service cinder] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:12.554132 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1577c2-ab20-42d1-a25f-9c3b37739537 service cinder] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:12.554504 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1577c2-ab20-42d1-a25f-9c3b37739537 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:12.565435 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d1577c2-ab20-42d1-a25f-9c3b37739537 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:12.566885 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d1577c2-ab20-42d1-a25f-9c3b37739537 service cinder] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:12.576363 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1379/2732] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:48:12 2022] GET /identity/v3/auth/tokens => generated 3276 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 25 04:48:13.020313 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ef062b0-8817-44c1-93ed-4d5ab5d0e37a service placement] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:13.022143 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-4ef062b0-8817-44c1-93ed-4d5ab5d0e37a service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:48:13.096865 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ef062b0-8817-44c1-93ed-4d5ab5d0e37a service placement] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:13.102057 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deddbcf3-fad5-4dfc-b9cb-5a244b5fbaab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:13.102690 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deddbcf3-fad5-4dfc-b9cb-5a244b5fbaab admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:13.102972 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deddbcf3-fad5-4dfc-b9cb-5a244b5fbaab admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:13.103248 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deddbcf3-fad5-4dfc-b9cb-5a244b5fbaab admin admin] PATH_INFO: `/v3/users/3c04eac386504f9f8adaf85f39454365` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:13.104402 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-deddbcf3-fad5-4dfc-b9cb-5a244b5fbaab admin admin] RBAC: Authorizing `identity:delete_user(user_id=3c04eac386504f9f8adaf85f39454365)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:13.106371 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-deddbcf3-fad5-4dfc-b9cb-5a244b5fbaab admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:13.152176 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-deddbcf3-fad5-4dfc-b9cb-5a244b5fbaab admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3c04eac386504f9f8adaf85f39454365'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:13.160348 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-deddbcf3-fad5-4dfc-b9cb-5a244b5fbaab admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3c04eac386504f9f8adaf85f39454365'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:13.175072 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-deddbcf3-fad5-4dfc-b9cb-5a244b5fbaab admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3c04eac386504f9f8adaf85f39454365'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:13.175718 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-deddbcf3-fad5-4dfc-b9cb-5a244b5fbaab admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3c04eac386504f9f8adaf85f39454365'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:13.182690 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-deddbcf3-fad5-4dfc-b9cb-5a244b5fbaab admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3c04eac386504f9f8adaf85f39454365'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:13.189025 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1354/2733] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:48:13 2022] DELETE /identity/v3/users/3c04eac386504f9f8adaf85f39454365 => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:48:13.361929 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d1577c2-ab20-42d1-a25f-9c3b37739537 service cinder] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:13.373290 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d1577c2-ab20-42d1-a25f-9c3b37739537 service cinder] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:13.431310 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63900c4b-5251-4d7c-adf5-e020ce86d92a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:13.431968 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63900c4b-5251-4d7c-adf5-e020ce86d92a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:13.432248 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63900c4b-5251-4d7c-adf5-e020ce86d92a admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:13.432519 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63900c4b-5251-4d7c-adf5-e020ce86d92a admin admin] PATH_INFO: `/v3/projects/e546577d458b44ef801876e47407a8b1` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:13.433664 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63900c4b-5251-4d7c-adf5-e020ce86d92a admin admin] RBAC: Authorizing `identity:delete_project(project_id=e546577d458b44ef801876e47407a8b1)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:13.435624 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63900c4b-5251-4d7c-adf5-e020ce86d92a admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:13.488095 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-63900c4b-5251-4d7c-adf5-e020ce86d92a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e546577d458b44ef801876e47407a8b1', 'request_id': 'req-63900c4b-5251-4d7c-adf5-e020ce86d92a'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:13.495427 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-63900c4b-5251-4d7c-adf5-e020ce86d92a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e546577d458b44ef801876e47407a8b1', 'request_id': 'req-63900c4b-5251-4d7c-adf5-e020ce86d92a'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:13.500929 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-63900c4b-5251-4d7c-adf5-e020ce86d92a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e546577d458b44ef801876e47407a8b1', 'request_id': 'req-63900c4b-5251-4d7c-adf5-e020ce86d92a'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:13.511732 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-63900c4b-5251-4d7c-adf5-e020ce86d92a admin admin] The token cache is being invalidate because project e546577d458b44ef801876e47407a8b1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81434) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:48:13.512024 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-63900c4b-5251-4d7c-adf5-e020ce86d92a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:13.512677 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-63900c4b-5251-4d7c-adf5-e020ce86d92a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:13.514825 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1380/2734] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:48:13 2022] DELETE /identity/v3/projects/e546577d458b44ef801876e47407a8b1 => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:48:13.518802 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deddbcf3-fad5-4dfc-b9cb-5a244b5fbaab admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:13.520234 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-deddbcf3-fad5-4dfc-b9cb-5a244b5fbaab admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:48:13.533458 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deddbcf3-fad5-4dfc-b9cb-5a244b5fbaab admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:13.559364 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:48:13.591787 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1983aa1-bb35-4721-aa81-e4e692fb9ae3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:13.592413 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1983aa1-bb35-4721-aa81-e4e692fb9ae3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:13.592728 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1983aa1-bb35-4721-aa81-e4e692fb9ae3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:13.593009 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1983aa1-bb35-4721-aa81-e4e692fb9ae3 admin admin] PATH_INFO: `/v3/users/275d12c86072479e8d958a84d5c4c7e8` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:13.594147 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1983aa1-bb35-4721-aa81-e4e692fb9ae3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=275d12c86072479e8d958a84d5c4c7e8)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:13.596095 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1983aa1-bb35-4721-aa81-e4e692fb9ae3 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:13.641920 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-c1983aa1-bb35-4721-aa81-e4e692fb9ae3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '275d12c86072479e8d958a84d5c4c7e8'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:13.649198 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-c1983aa1-bb35-4721-aa81-e4e692fb9ae3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '275d12c86072479e8d958a84d5c4c7e8'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:13.661242 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-c1983aa1-bb35-4721-aa81-e4e692fb9ae3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '275d12c86072479e8d958a84d5c4c7e8'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:13.661883 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-c1983aa1-bb35-4721-aa81-e4e692fb9ae3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '275d12c86072479e8d958a84d5c4c7e8'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:13.668713 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-c1983aa1-bb35-4721-aa81-e4e692fb9ae3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '275d12c86072479e8d958a84d5c4c7e8'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:13.675137 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1355/2735] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:48:13 2022] DELETE /identity/v3/users/275d12c86072479e8d958a84d5c4c7e8 => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:48:13.846566 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63900c4b-5251-4d7c-adf5-e020ce86d92a admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:13.848077 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-63900c4b-5251-4d7c-adf5-e020ce86d92a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:48:13.861906 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63900c4b-5251-4d7c-adf5-e020ce86d92a admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:13.910204 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:48:13.922046 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-707d15ea-6992-4949-b84a-9dde84ed2d67 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:13.922684 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-707d15ea-6992-4949-b84a-9dde84ed2d67 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:13.922964 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-707d15ea-6992-4949-b84a-9dde84ed2d67 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:13.923240 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-707d15ea-6992-4949-b84a-9dde84ed2d67 admin admin] PATH_INFO: `/v3/projects/a3ad197b14d64415a70f9da43c71f34a` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:13.924357 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-707d15ea-6992-4949-b84a-9dde84ed2d67 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a3ad197b14d64415a70f9da43c71f34a)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:13.926297 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-707d15ea-6992-4949-b84a-9dde84ed2d67 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:13.971671 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-707d15ea-6992-4949-b84a-9dde84ed2d67 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a3ad197b14d64415a70f9da43c71f34a', 'request_id': 'req-707d15ea-6992-4949-b84a-9dde84ed2d67'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:13.978993 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-707d15ea-6992-4949-b84a-9dde84ed2d67 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a3ad197b14d64415a70f9da43c71f34a', 'request_id': 'req-707d15ea-6992-4949-b84a-9dde84ed2d67'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:13.985737 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-707d15ea-6992-4949-b84a-9dde84ed2d67 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a3ad197b14d64415a70f9da43c71f34a', 'request_id': 'req-707d15ea-6992-4949-b84a-9dde84ed2d67'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:13.996519 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-707d15ea-6992-4949-b84a-9dde84ed2d67 admin admin] The token cache is being invalidate because project a3ad197b14d64415a70f9da43c71f34a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81434) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:48:13.996857 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-707d15ea-6992-4949-b84a-9dde84ed2d67 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:13.997468 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-707d15ea-6992-4949-b84a-9dde84ed2d67 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:13.999177 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1381/2736] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:48:13 2022] DELETE /identity/v3/projects/a3ad197b14d64415a70f9da43c71f34a => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:48:14.004142 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c42d1ac-4083-4b0f-9217-c17f45c37a76 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:14.004432 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c42d1ac-4083-4b0f-9217-c17f45c37a76 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:14.004750 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c42d1ac-4083-4b0f-9217-c17f45c37a76 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:14.025204 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-5c42d1ac-4083-4b0f-9217-c17f45c37a76 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:48:14.026421 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-5c42d1ac-4083-4b0f-9217-c17f45c37a76 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:48:14.086791 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1356/2737] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:48:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:48:14.095305 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b062eaa9-34ae-40a6-9e3c-8450dd329110 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:14.095687 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b062eaa9-34ae-40a6-9e3c-8450dd329110 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:14.095968 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b062eaa9-34ae-40a6-9e3c-8450dd329110 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:14.116463 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-b062eaa9-34ae-40a6-9e3c-8450dd329110 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:48:14.117709 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-b062eaa9-34ae-40a6-9e3c-8450dd329110 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:48:14.126136 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1382/2738] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:48:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:48:14.130711 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c42d1ac-4083-4b0f-9217-c17f45c37a76 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:14.140387 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c42d1ac-4083-4b0f-9217-c17f45c37a76 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:14.145141 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a7daf62-d6cd-44f1-9073-ff23aa68dd06 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:14.145747 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a7daf62-d6cd-44f1-9073-ff23aa68dd06 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:14.146028 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a7daf62-d6cd-44f1-9073-ff23aa68dd06 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:14.146303 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a7daf62-d6cd-44f1-9073-ff23aa68dd06 admin admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:14.146966 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a7daf62-d6cd-44f1-9073-ff23aa68dd06 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:48:14.147273 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a7daf62-d6cd-44f1-9073-ff23aa68dd06 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:14.148943 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a7daf62-d6cd-44f1-9073-ff23aa68dd06 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:14.165526 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1357/2739] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Sep 25 04:48:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:48:14.166577 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b062eaa9-34ae-40a6-9e3c-8450dd329110 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:14.177115 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b062eaa9-34ae-40a6-9e3c-8450dd329110 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:14.182707 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d9e53e7-4bca-46e5-9de1-1b392151709d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:14.183314 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d9e53e7-4bca-46e5-9de1-1b392151709d admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:14.183664 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d9e53e7-4bca-46e5-9de1-1b392151709d admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:14.184050 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d9e53e7-4bca-46e5-9de1-1b392151709d admin admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:14.184723 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d9e53e7-4bca-46e5-9de1-1b392151709d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:14.187102 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d9e53e7-4bca-46e5-9de1-1b392151709d admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:14.217997 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1383/2740] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:48:14 2022] POST /identity/v3/projects => generated 381 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 04:48:14.222324 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a7daf62-d6cd-44f1-9073-ff23aa68dd06 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:14.238613 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a7daf62-d6cd-44f1-9073-ff23aa68dd06 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:14.278531 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:48:14.307842 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83a2ea6b-eb44-4d71-979a-c366b5a8523a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:14.308486 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83a2ea6b-eb44-4d71-979a-c366b5a8523a admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:14.308806 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83a2ea6b-eb44-4d71-979a-c366b5a8523a admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:14.309104 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83a2ea6b-eb44-4d71-979a-c366b5a8523a admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:14.309740 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83a2ea6b-eb44-4d71-979a-c366b5a8523a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:14.311843 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83a2ea6b-eb44-4d71-979a-c366b5a8523a admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:14.333767 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1358/2741] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:48:14 2022] POST /identity/v3/users => generated 291 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 04:48:14.337824 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d9e53e7-4bca-46e5-9de1-1b392151709d admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:14.349152 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d9e53e7-4bca-46e5-9de1-1b392151709d admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:14.353965 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-659edffa-f050-46c5-bedf-58996ae0b93f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:14.354565 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-659edffa-f050-46c5-bedf-58996ae0b93f admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:14.354847 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-659edffa-f050-46c5-bedf-58996ae0b93f admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:14.355119 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-659edffa-f050-46c5-bedf-58996ae0b93f admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:14.355750 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-659edffa-f050-46c5-bedf-58996ae0b93f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:14.357410 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-659edffa-f050-46c5-bedf-58996ae0b93f admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:14.367130 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1384/2742] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:48:14 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:48:14.371000 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83a2ea6b-eb44-4d71-979a-c366b5a8523a admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:14.381111 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83a2ea6b-eb44-4d71-979a-c366b5a8523a admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:14.386054 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a546081-c98b-494e-9b6e-956fc1b639ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:14.386645 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a546081-c98b-494e-9b6e-956fc1b639ae admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:14.386925 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a546081-c98b-494e-9b6e-956fc1b639ae admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:14.387197 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a546081-c98b-494e-9b6e-956fc1b639ae admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:14.387828 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a546081-c98b-494e-9b6e-956fc1b639ae admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:14.390125 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a546081-c98b-494e-9b6e-956fc1b639ae admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:14.400892 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1359/2743] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:48:14 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:48:14.405633 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-659edffa-f050-46c5-bedf-58996ae0b93f admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:14.415608 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-659edffa-f050-46c5-bedf-58996ae0b93f admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:14.420178 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bf6313b-f218-4869-81cd-cf6979faac2d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:14.420862 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bf6313b-f218-4869-81cd-cf6979faac2d admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:14.421144 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bf6313b-f218-4869-81cd-cf6979faac2d admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:14.421418 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bf6313b-f218-4869-81cd-cf6979faac2d admin admin] PATH_INFO: `/v3/projects/313eed2ef1484803afa88c4b4c5020f2/users/19b7d29d1f0349ecbd6cf1cf9d8ce816/roles/8f6cc574056c4278bc9778f14dc4b411` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:14.435621 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bf6313b-f218-4869-81cd-cf6979faac2d admin admin] RBAC: Authorizing `identity:create_grant(project_id=313eed2ef1484803afa88c4b4c5020f2, user_id=19b7d29d1f0349ecbd6cf1cf9d8ce816, role_id=8f6cc574056c4278bc9778f14dc4b411)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:14.438301 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bf6313b-f218-4869-81cd-cf6979faac2d admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:14.451891 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1385/2744] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:48:14 2022] PUT /identity/v3/projects/313eed2ef1484803afa88c4b4c5020f2/users/19b7d29d1f0349ecbd6cf1cf9d8ce816/roles/8f6cc574056c4278bc9778f14dc4b411 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 04:48:14.463229 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8de41153-27d0-48d0-8d2c-695acedcc2d5 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:14.463517 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8de41153-27d0-48d0-8d2c-695acedcc2d5 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:14.463796 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8de41153-27d0-48d0-8d2c-695acedcc2d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:14.492468 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-8de41153-27d0-48d0-8d2c-695acedcc2d5 None None] MFA Rules not processed for user `19b7d29d1f0349ecbd6cf1cf9d8ce816`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:48:14.493715 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-8de41153-27d0-48d0-8d2c-695acedcc2d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:48:14.557126 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1360/2745] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:48:14 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:48:14.568700 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bf6313b-f218-4869-81cd-cf6979faac2d admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:14.570386 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-3bf6313b-f218-4869-81cd-cf6979faac2d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:48:14.629605 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bf6313b-f218-4869-81cd-cf6979faac2d admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:14.633234 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05bdee61-0537-4274-8c0e-9a5ddc23c539 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ffbc136c77b84e8c87128f7247f4ec03', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:14.633844 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05bdee61-0537-4274-8c0e-9a5ddc23c539 service cinder] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:14.634125 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05bdee61-0537-4274-8c0e-9a5ddc23c539 service cinder] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:14.634400 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05bdee61-0537-4274-8c0e-9a5ddc23c539 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:14.643163 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05bdee61-0537-4274-8c0e-9a5ddc23c539 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:14.644582 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05bdee61-0537-4274-8c0e-9a5ddc23c539 service cinder] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:14.653309 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1386/2746] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:48:14 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:48:14.672141 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8de41153-27d0-48d0-8d2c-695acedcc2d5 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:14.734143 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8de41153-27d0-48d0-8d2c-695acedcc2d5 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:14.738757 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f7e4600-f03a-424f-9140-df586ea4cbe6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:14.739378 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f7e4600-f03a-424f-9140-df586ea4cbe6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:14.739659 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f7e4600-f03a-424f-9140-df586ea4cbe6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:14.739933 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f7e4600-f03a-424f-9140-df586ea4cbe6 admin admin] PATH_INFO: `/v3/users/19b7d29d1f0349ecbd6cf1cf9d8ce816` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:14.741141 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f7e4600-f03a-424f-9140-df586ea4cbe6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=19b7d29d1f0349ecbd6cf1cf9d8ce816)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:14.743051 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f7e4600-f03a-424f-9140-df586ea4cbe6 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:14.788379 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=90,keystone:INSERT=3,keystone:DELETE=7 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:48:14.790296 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-2f7e4600-f03a-424f-9140-df586ea4cbe6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '19b7d29d1f0349ecbd6cf1cf9d8ce816'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:14.798478 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-2f7e4600-f03a-424f-9140-df586ea4cbe6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '19b7d29d1f0349ecbd6cf1cf9d8ce816'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:14.811382 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-2f7e4600-f03a-424f-9140-df586ea4cbe6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '19b7d29d1f0349ecbd6cf1cf9d8ce816'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:14.812037 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-2f7e4600-f03a-424f-9140-df586ea4cbe6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '19b7d29d1f0349ecbd6cf1cf9d8ce816'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:14.818764 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-2f7e4600-f03a-424f-9140-df586ea4cbe6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '19b7d29d1f0349ecbd6cf1cf9d8ce816'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:14.824532 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1361/2747] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:48:14 2022] DELETE /identity/v3/users/19b7d29d1f0349ecbd6cf1cf9d8ce816 => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:48:14.829745 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bef8c44-773d-4c7b-8b5c-af0a2d38da2c None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:14.830029 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bef8c44-773d-4c7b-8b5c-af0a2d38da2c None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:14.830301 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bef8c44-773d-4c7b-8b5c-af0a2d38da2c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:14.849242 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-3bef8c44-773d-4c7b-8b5c-af0a2d38da2c None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:48:14.850428 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-3bef8c44-773d-4c7b-8b5c-af0a2d38da2c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:48:14.863166 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:48:14.910881 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1387/2748] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:48:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:48:14.920623 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f7e4600-f03a-424f-9140-df586ea4cbe6 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:14.922057 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-2f7e4600-f03a-424f-9140-df586ea4cbe6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:48:14.979627 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f7e4600-f03a-424f-9140-df586ea4cbe6 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:14.983560 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a414594-19c1-4aab-8227-1e6217dcd8c7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:14.984163 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a414594-19c1-4aab-8227-1e6217dcd8c7 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:14.984446 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a414594-19c1-4aab-8227-1e6217dcd8c7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:14.984750 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a414594-19c1-4aab-8227-1e6217dcd8c7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:14.993736 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a414594-19c1-4aab-8227-1e6217dcd8c7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:14.995150 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a414594-19c1-4aab-8227-1e6217dcd8c7 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:15.005663 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1362/2749] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:48:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:48:15.280271 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bef8c44-773d-4c7b-8b5c-af0a2d38da2c None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:15.281742 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-3bef8c44-773d-4c7b-8b5c-af0a2d38da2c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:48:15.297806 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bef8c44-773d-4c7b-8b5c-af0a2d38da2c None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:15.313389 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59b598fb-a8a2-4aea-bfc6-1e71fb4cabe8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:15.314033 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59b598fb-a8a2-4aea-bfc6-1e71fb4cabe8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:15.314317 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59b598fb-a8a2-4aea-bfc6-1e71fb4cabe8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:15.314598 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59b598fb-a8a2-4aea-bfc6-1e71fb4cabe8 admin admin] PATH_INFO: `/v3/projects/313eed2ef1484803afa88c4b4c5020f2` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:15.315748 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59b598fb-a8a2-4aea-bfc6-1e71fb4cabe8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=313eed2ef1484803afa88c4b4c5020f2)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:15.317741 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59b598fb-a8a2-4aea-bfc6-1e71fb4cabe8 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:15.365529 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-59b598fb-a8a2-4aea-bfc6-1e71fb4cabe8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '313eed2ef1484803afa88c4b4c5020f2', 'request_id': 'req-59b598fb-a8a2-4aea-bfc6-1e71fb4cabe8'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:15.375603 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-59b598fb-a8a2-4aea-bfc6-1e71fb4cabe8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '313eed2ef1484803afa88c4b4c5020f2', 'request_id': 'req-59b598fb-a8a2-4aea-bfc6-1e71fb4cabe8'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:15.381385 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-59b598fb-a8a2-4aea-bfc6-1e71fb4cabe8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '313eed2ef1484803afa88c4b4c5020f2', 'request_id': 'req-59b598fb-a8a2-4aea-bfc6-1e71fb4cabe8'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:15.392577 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-59b598fb-a8a2-4aea-bfc6-1e71fb4cabe8 admin admin] The token cache is being invalidate because project 313eed2ef1484803afa88c4b4c5020f2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81434) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:48:15.392911 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-59b598fb-a8a2-4aea-bfc6-1e71fb4cabe8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:15.393577 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-59b598fb-a8a2-4aea-bfc6-1e71fb4cabe8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:15.395057 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1388/2750] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:48:15 2022] DELETE /identity/v3/projects/313eed2ef1484803afa88c4b4c5020f2 => generated 0 bytes in 115 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:48:15.399911 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c528b602-b250-4937-b52b-56918fdbb273 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:15.400196 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c528b602-b250-4937-b52b-56918fdbb273 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:15.400494 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c528b602-b250-4937-b52b-56918fdbb273 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:15.420682 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-c528b602-b250-4937-b52b-56918fdbb273 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:48:15.421875 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-c528b602-b250-4937-b52b-56918fdbb273 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:48:15.480654 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1363/2751] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:48:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:48:15.488835 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4184e6b-3c79-4bfd-a03a-e9dfaa0c0800 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:15.489125 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4184e6b-3c79-4bfd-a03a-e9dfaa0c0800 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:15.489400 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4184e6b-3c79-4bfd-a03a-e9dfaa0c0800 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:15.508774 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-c4184e6b-3c79-4bfd-a03a-e9dfaa0c0800 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:48:15.509963 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-c4184e6b-3c79-4bfd-a03a-e9dfaa0c0800 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:48:15.518014 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1389/2752] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:48:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:48:15.522071 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c528b602-b250-4937-b52b-56918fdbb273 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:15.532221 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c528b602-b250-4937-b52b-56918fdbb273 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:15.536814 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3682cd5f-9a7c-4b20-9563-41ed337aee56 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:15.537403 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3682cd5f-9a7c-4b20-9563-41ed337aee56 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:15.537679 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3682cd5f-9a7c-4b20-9563-41ed337aee56 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:15.537948 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3682cd5f-9a7c-4b20-9563-41ed337aee56 admin admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:15.538578 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3682cd5f-9a7c-4b20-9563-41ed337aee56 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:48:15.538878 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3682cd5f-9a7c-4b20-9563-41ed337aee56 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:15.540453 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3682cd5f-9a7c-4b20-9563-41ed337aee56 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:15.552197 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1364/2753] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Sep 25 04:48:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:48:15.556173 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4184e6b-3c79-4bfd-a03a-e9dfaa0c0800 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:15.565955 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4184e6b-3c79-4bfd-a03a-e9dfaa0c0800 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:15.570557 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-269eeff4-a675-45ab-8ff9-2128d2580df5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:15.571147 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-269eeff4-a675-45ab-8ff9-2128d2580df5 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:15.571422 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-269eeff4-a675-45ab-8ff9-2128d2580df5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:15.571692 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-269eeff4-a675-45ab-8ff9-2128d2580df5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:15.572321 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-269eeff4-a675-45ab-8ff9-2128d2580df5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:15.574222 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-269eeff4-a675-45ab-8ff9-2128d2580df5 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:15.601276 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1390/2754] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:48:15 2022] POST /identity/v3/projects => generated 375 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:48:15.605471 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3682cd5f-9a7c-4b20-9563-41ed337aee56 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:15.615823 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3682cd5f-9a7c-4b20-9563-41ed337aee56 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:15.663735 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=96,keystone:INSERT=1,keystone:DELETE=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:48:15.674374 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e57dc9ce-9315-48d9-b6c6-efa353b17f82 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:15.674974 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e57dc9ce-9315-48d9-b6c6-efa353b17f82 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:15.675255 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e57dc9ce-9315-48d9-b6c6-efa353b17f82 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:15.675525 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e57dc9ce-9315-48d9-b6c6-efa353b17f82 admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:15.676189 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e57dc9ce-9315-48d9-b6c6-efa353b17f82 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:15.678170 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e57dc9ce-9315-48d9-b6c6-efa353b17f82 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:15.699817 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1365/2755] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:48:15 2022] POST /identity/v3/users => generated 288 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:48:15.704821 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-269eeff4-a675-45ab-8ff9-2128d2580df5 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:15.715310 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-269eeff4-a675-45ab-8ff9-2128d2580df5 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:15.719885 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d3293f2-ab32-44df-860b-f863b3a97c60 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:15.720474 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d3293f2-ab32-44df-860b-f863b3a97c60 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:15.720790 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d3293f2-ab32-44df-860b-f863b3a97c60 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:15.721062 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d3293f2-ab32-44df-860b-f863b3a97c60 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:15.721699 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d3293f2-ab32-44df-860b-f863b3a97c60 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:15.723272 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d3293f2-ab32-44df-860b-f863b3a97c60 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:15.733130 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1391/2756] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:48:15 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:48:15.736890 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e57dc9ce-9315-48d9-b6c6-efa353b17f82 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:15.746818 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e57dc9ce-9315-48d9-b6c6-efa353b17f82 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:15.752245 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d86f9c1-7d0e-470f-aae5-369f14ca1272 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:15.752870 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d86f9c1-7d0e-470f-aae5-369f14ca1272 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:15.753152 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d86f9c1-7d0e-470f-aae5-369f14ca1272 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:15.753425 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d86f9c1-7d0e-470f-aae5-369f14ca1272 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:15.754054 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d86f9c1-7d0e-470f-aae5-369f14ca1272 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:15.755645 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d86f9c1-7d0e-470f-aae5-369f14ca1272 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:15.771832 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1366/2757] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:48:15 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:48:15.772790 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d3293f2-ab32-44df-860b-f863b3a97c60 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:15.785818 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d3293f2-ab32-44df-860b-f863b3a97c60 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:15.790721 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43d4da7c-3ec7-4f36-83bb-eac7ca8ef880 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:15.791374 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d4da7c-3ec7-4f36-83bb-eac7ca8ef880 admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:15.791650 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d4da7c-3ec7-4f36-83bb-eac7ca8ef880 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:15.792010 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d4da7c-3ec7-4f36-83bb-eac7ca8ef880 admin admin] PATH_INFO: `/v3/projects/ca4031a419ae4e8a8f82cbadef2b1fd1/users/f5dd109b60f1416280da073ba231c211/roles/8f6cc574056c4278bc9778f14dc4b411` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:15.806850 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43d4da7c-3ec7-4f36-83bb-eac7ca8ef880 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ca4031a419ae4e8a8f82cbadef2b1fd1, user_id=f5dd109b60f1416280da073ba231c211, role_id=8f6cc574056c4278bc9778f14dc4b411)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:15.809509 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43d4da7c-3ec7-4f36-83bb-eac7ca8ef880 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:15.822658 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1392/2758] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:48:15 2022] PUT /identity/v3/projects/ca4031a419ae4e8a8f82cbadef2b1fd1/users/f5dd109b60f1416280da073ba231c211/roles/8f6cc574056c4278bc9778f14dc4b411 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 04:48:15.833028 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e1b8090-a7e7-4a6c-9d15-ee7bf9de8e4d None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:15.833328 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e1b8090-a7e7-4a6c-9d15-ee7bf9de8e4d None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:15.833588 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e1b8090-a7e7-4a6c-9d15-ee7bf9de8e4d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:15.861840 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-2e1b8090-a7e7-4a6c-9d15-ee7bf9de8e4d None None] MFA Rules not processed for user `f5dd109b60f1416280da073ba231c211`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:48:15.863059 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-2e1b8090-a7e7-4a6c-9d15-ee7bf9de8e4d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:48:15.926994 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1367/2759] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:48:15 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:48:15.938851 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43d4da7c-3ec7-4f36-83bb-eac7ca8ef880 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:15.940591 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-43d4da7c-3ec7-4f36-83bb-eac7ca8ef880 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:48:15.983907 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:48:15.999918 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43d4da7c-3ec7-4f36-83bb-eac7ca8ef880 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:16.003622 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87cceaed-a434-4c32-8540-c4d4e799c197 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ffbc136c77b84e8c87128f7247f4ec03', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:16.004227 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87cceaed-a434-4c32-8540-c4d4e799c197 service cinder] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:16.004508 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87cceaed-a434-4c32-8540-c4d4e799c197 service cinder] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:16.004832 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87cceaed-a434-4c32-8540-c4d4e799c197 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:16.013071 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87cceaed-a434-4c32-8540-c4d4e799c197 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:16.014663 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87cceaed-a434-4c32-8540-c4d4e799c197 service cinder] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:16.023360 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1393/2760] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:48:15 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:48:20.755871 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e1b8090-a7e7-4a6c-9d15-ee7bf9de8e4d None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:20.757306 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-2e1b8090-a7e7-4a6c-9d15-ee7bf9de8e4d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:48:20.821169 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e1b8090-a7e7-4a6c-9d15-ee7bf9de8e4d None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:20.826427 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07a33630-f832-43fc-8721-4b2c0604f064 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:20.827058 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07a33630-f832-43fc-8721-4b2c0604f064 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:20.827339 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07a33630-f832-43fc-8721-4b2c0604f064 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:20.827615 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07a33630-f832-43fc-8721-4b2c0604f064 admin admin] PATH_INFO: `/v3/users/fb87a4a326b6420d8b93784ec0def9f7` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:20.828763 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07a33630-f832-43fc-8721-4b2c0604f064 admin admin] RBAC: Authorizing `identity:delete_user(user_id=fb87a4a326b6420d8b93784ec0def9f7)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:20.830670 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07a33630-f832-43fc-8721-4b2c0604f064 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:20.878671 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-07a33630-f832-43fc-8721-4b2c0604f064 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fb87a4a326b6420d8b93784ec0def9f7'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:20.886155 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-07a33630-f832-43fc-8721-4b2c0604f064 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fb87a4a326b6420d8b93784ec0def9f7'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:20.894888 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=87,keystone:INSERT=4,keystone:DELETE=9 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:48:20.901543 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-07a33630-f832-43fc-8721-4b2c0604f064 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fb87a4a326b6420d8b93784ec0def9f7'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:20.902658 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-07a33630-f832-43fc-8721-4b2c0604f064 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fb87a4a326b6420d8b93784ec0def9f7'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:20.910158 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-07a33630-f832-43fc-8721-4b2c0604f064 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fb87a4a326b6420d8b93784ec0def9f7'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:20.916065 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1368/2761] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:48:20 2022] DELETE /identity/v3/users/fb87a4a326b6420d8b93784ec0def9f7 => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:48:20.920477 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3772d117-a991-4a66-a6e7-d06d05919eca None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:20.920989 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3772d117-a991-4a66-a6e7-d06d05919eca None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:20.921109 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3772d117-a991-4a66-a6e7-d06d05919eca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:20.940227 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-3772d117-a991-4a66-a6e7-d06d05919eca None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:48:20.941478 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-3772d117-a991-4a66-a6e7-d06d05919eca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:48:20.999593 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1394/2762] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:48:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:48:21.009388 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07a33630-f832-43fc-8721-4b2c0604f064 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:21.010808 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-07a33630-f832-43fc-8721-4b2c0604f064 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:48:21.069784 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07a33630-f832-43fc-8721-4b2c0604f064 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:21.073495 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9634e7ec-fdc1-419e-8c0f-8b2d6a16f939 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:21.074104 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9634e7ec-fdc1-419e-8c0f-8b2d6a16f939 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:21.074387 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9634e7ec-fdc1-419e-8c0f-8b2d6a16f939 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:21.074664 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9634e7ec-fdc1-419e-8c0f-8b2d6a16f939 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:21.084189 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9634e7ec-fdc1-419e-8c0f-8b2d6a16f939 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:21.085647 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9634e7ec-fdc1-419e-8c0f-8b2d6a16f939 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:21.096345 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1369/2763] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:48:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:48:21.366950 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3772d117-a991-4a66-a6e7-d06d05919eca None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:21.368513 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-3772d117-a991-4a66-a6e7-d06d05919eca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:48:21.382704 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3772d117-a991-4a66-a6e7-d06d05919eca None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:21.387478 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7005147d-4552-4dc7-ba0a-01dba88b0879 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:21.388382 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7005147d-4552-4dc7-ba0a-01dba88b0879 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:21.390020 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7005147d-4552-4dc7-ba0a-01dba88b0879 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:21.390020 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7005147d-4552-4dc7-ba0a-01dba88b0879 admin admin] PATH_INFO: `/v3/projects/df5dfe4898574c00ab5e51ccbb027be6` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:21.391149 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7005147d-4552-4dc7-ba0a-01dba88b0879 admin admin] RBAC: Authorizing `identity:delete_project(project_id=df5dfe4898574c00ab5e51ccbb027be6)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:21.393147 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7005147d-4552-4dc7-ba0a-01dba88b0879 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:21.441913 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-7005147d-4552-4dc7-ba0a-01dba88b0879 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'df5dfe4898574c00ab5e51ccbb027be6', 'request_id': 'req-7005147d-4552-4dc7-ba0a-01dba88b0879'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:21.448985 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-7005147d-4552-4dc7-ba0a-01dba88b0879 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'df5dfe4898574c00ab5e51ccbb027be6', 'request_id': 'req-7005147d-4552-4dc7-ba0a-01dba88b0879'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:21.454740 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-7005147d-4552-4dc7-ba0a-01dba88b0879 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'df5dfe4898574c00ab5e51ccbb027be6', 'request_id': 'req-7005147d-4552-4dc7-ba0a-01dba88b0879'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:21.466043 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-7005147d-4552-4dc7-ba0a-01dba88b0879 admin admin] The token cache is being invalidate because project df5dfe4898574c00ab5e51ccbb027be6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81434) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:48:21.466899 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-7005147d-4552-4dc7-ba0a-01dba88b0879 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:21.467551 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-7005147d-4552-4dc7-ba0a-01dba88b0879 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:21.469070 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1395/2764] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:48:21 2022] DELETE /identity/v3/projects/df5dfe4898574c00ab5e51ccbb027be6 => generated 0 bytes in 103 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:48:31.091338 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=34,keystone:DELETE=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:48:31.462693 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=62,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:48:36.499830 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9634e7ec-fdc1-419e-8c0f-8b2d6a16f939 service neutron] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:36.509019 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9634e7ec-fdc1-419e-8c0f-8b2d6a16f939 service neutron] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:36.558637 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fab5736-33a5-4eae-93c8-a383c3902cec service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:36.559241 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fab5736-33a5-4eae-93c8-a383c3902cec service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:36.559522 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fab5736-33a5-4eae-93c8-a383c3902cec service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:36.559796 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fab5736-33a5-4eae-93c8-a383c3902cec service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:36.561614 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-9fab5736-33a5-4eae-93c8-a383c3902cec service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:48:36.624096 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fab5736-33a5-4eae-93c8-a383c3902cec service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:36.625556 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fab5736-33a5-4eae-93c8-a383c3902cec service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:36.635553 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1370/2765] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:48:36 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 136 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:48:39.814265 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7005147d-4552-4dc7-ba0a-01dba88b0879 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:39.815719 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-7005147d-4552-4dc7-ba0a-01dba88b0879 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:48:39.829092 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7005147d-4552-4dc7-ba0a-01dba88b0879 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:39.833742 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a920ee8-4b5a-46b7-8ce0-e6047e36b068 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:39.834367 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a920ee8-4b5a-46b7-8ce0-e6047e36b068 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:39.834642 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a920ee8-4b5a-46b7-8ce0-e6047e36b068 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:39.834908 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a920ee8-4b5a-46b7-8ce0-e6047e36b068 admin admin] PATH_INFO: `/v3/users/e7354aa144df4ad69cd7d12f11b1d8a5` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:39.836016 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a920ee8-4b5a-46b7-8ce0-e6047e36b068 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e7354aa144df4ad69cd7d12f11b1d8a5)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:39.837946 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a920ee8-4b5a-46b7-8ce0-e6047e36b068 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:39.884060 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-4a920ee8-4b5a-46b7-8ce0-e6047e36b068 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e7354aa144df4ad69cd7d12f11b1d8a5'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:39.892039 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-4a920ee8-4b5a-46b7-8ce0-e6047e36b068 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e7354aa144df4ad69cd7d12f11b1d8a5'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:39.903678 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-4a920ee8-4b5a-46b7-8ce0-e6047e36b068 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e7354aa144df4ad69cd7d12f11b1d8a5'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:39.904336 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-4a920ee8-4b5a-46b7-8ce0-e6047e36b068 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e7354aa144df4ad69cd7d12f11b1d8a5'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:39.911364 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-4a920ee8-4b5a-46b7-8ce0-e6047e36b068 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e7354aa144df4ad69cd7d12f11b1d8a5'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:39.917447 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1396/2766] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:48:39 2022] DELETE /identity/v3/users/e7354aa144df4ad69cd7d12f11b1d8a5 => generated 0 bytes in 105 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:48:40.111723 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fab5736-33a5-4eae-93c8-a383c3902cec service neutron] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:40.116083 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a920ee8-4b5a-46b7-8ce0-e6047e36b068 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:40.117568 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-4a920ee8-4b5a-46b7-8ce0-e6047e36b068 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:48:40.130246 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fab5736-33a5-4eae-93c8-a383c3902cec service neutron] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:40.191532 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02a5e250-9a11-425d-a9c2-843912f41298 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:40.192168 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02a5e250-9a11-425d-a9c2-843912f41298 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:40.192357 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a920ee8-4b5a-46b7-8ce0-e6047e36b068 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:40.192481 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02a5e250-9a11-425d-a9c2-843912f41298 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:40.192746 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02a5e250-9a11-425d-a9c2-843912f41298 admin admin] PATH_INFO: `/v3/projects/a2ac515b97884375b3e85c6e54de81bc` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:40.193899 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02a5e250-9a11-425d-a9c2-843912f41298 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a2ac515b97884375b3e85c6e54de81bc)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:40.195816 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02a5e250-9a11-425d-a9c2-843912f41298 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:40.197074 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29fe846d-98c7-467f-8476-de5f71115dda admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:40.197709 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29fe846d-98c7-467f-8476-de5f71115dda admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:40.197983 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29fe846d-98c7-467f-8476-de5f71115dda admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:40.198262 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29fe846d-98c7-467f-8476-de5f71115dda admin admin] PATH_INFO: `/v3/users/f5dd109b60f1416280da073ba231c211` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:40.199391 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29fe846d-98c7-467f-8476-de5f71115dda admin admin] RBAC: Authorizing `identity:delete_user(user_id=f5dd109b60f1416280da073ba231c211)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:40.201334 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29fe846d-98c7-467f-8476-de5f71115dda admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:40.246428 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-02a5e250-9a11-425d-a9c2-843912f41298 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a2ac515b97884375b3e85c6e54de81bc', 'request_id': 'req-02a5e250-9a11-425d-a9c2-843912f41298'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:40.248995 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=95,keystone:DELETE=4,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:48:40.252378 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-29fe846d-98c7-467f-8476-de5f71115dda admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f5dd109b60f1416280da073ba231c211'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:40.255377 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-02a5e250-9a11-425d-a9c2-843912f41298 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a2ac515b97884375b3e85c6e54de81bc', 'request_id': 'req-02a5e250-9a11-425d-a9c2-843912f41298'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:40.260849 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-29fe846d-98c7-467f-8476-de5f71115dda admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f5dd109b60f1416280da073ba231c211'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:40.263308 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-02a5e250-9a11-425d-a9c2-843912f41298 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a2ac515b97884375b3e85c6e54de81bc', 'request_id': 'req-02a5e250-9a11-425d-a9c2-843912f41298'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:40.274469 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-29fe846d-98c7-467f-8476-de5f71115dda admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f5dd109b60f1416280da073ba231c211'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:40.275135 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-29fe846d-98c7-467f-8476-de5f71115dda admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f5dd109b60f1416280da073ba231c211'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:40.275471 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-02a5e250-9a11-425d-a9c2-843912f41298 admin admin] The token cache is being invalidate because project a2ac515b97884375b3e85c6e54de81bc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:48:40.275764 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-02a5e250-9a11-425d-a9c2-843912f41298 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:40.276549 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-02a5e250-9a11-425d-a9c2-843912f41298 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:40.277205 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=78,keystone:DELETE=20,keystone:INSERT=2 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:48:40.278213 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1371/2767] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:48:40 2022] DELETE /identity/v3/projects/a2ac515b97884375b3e85c6e54de81bc => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:48:40.283198 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f53e1a1f-5ddd-46f9-a03c-14c107349cae None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:40.283502 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f53e1a1f-5ddd-46f9-a03c-14c107349cae None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:40.283790 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f53e1a1f-5ddd-46f9-a03c-14c107349cae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:40.284536 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-29fe846d-98c7-467f-8476-de5f71115dda admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f5dd109b60f1416280da073ba231c211'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:40.291940 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1397/2768] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:48:40 2022] DELETE /identity/v3/users/f5dd109b60f1416280da073ba231c211 => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:48:40.296031 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d69c60c6-e690-42bd-bca0-07e591171ccf None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:40.296350 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d69c60c6-e690-42bd-bca0-07e591171ccf None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:40.296675 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d69c60c6-e690-42bd-bca0-07e591171ccf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:40.306282 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-f53e1a1f-5ddd-46f9-a03c-14c107349cae None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:48:40.307493 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-f53e1a1f-5ddd-46f9-a03c-14c107349cae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:48:40.316237 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-d69c60c6-e690-42bd-bca0-07e591171ccf None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:48:40.317475 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-d69c60c6-e690-42bd-bca0-07e591171ccf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:48:40.371439 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1372/2769] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:48:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 04:48:40.379903 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1398/2770] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:48:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 04:48:40.381197 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb78027d-dd41-4a7c-b94f-a6463b32b2a7 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:40.381487 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb78027d-dd41-4a7c-b94f-a6463b32b2a7 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:40.381764 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb78027d-dd41-4a7c-b94f-a6463b32b2a7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:40.389223 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d69c60c6-e690-42bd-bca0-07e591171ccf None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:40.391189 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-d69c60c6-e690-42bd-bca0-07e591171ccf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:48:40.404175 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-bb78027d-dd41-4a7c-b94f-a6463b32b2a7 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:48:40.405448 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-bb78027d-dd41-4a7c-b94f-a6463b32b2a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:48:40.413522 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1373/2771] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:48:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 04:48:40.417743 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb78027d-dd41-4a7c-b94f-a6463b32b2a7 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:40.427644 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb78027d-dd41-4a7c-b94f-a6463b32b2a7 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:40.433814 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a6c9498-a83f-46ae-9b4a-41878d8f004c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:40.434403 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a6c9498-a83f-46ae-9b4a-41878d8f004c admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:40.435044 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a6c9498-a83f-46ae-9b4a-41878d8f004c admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:40.435318 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a6c9498-a83f-46ae-9b4a-41878d8f004c admin admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:40.435959 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a6c9498-a83f-46ae-9b4a-41878d8f004c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:48:40.436256 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a6c9498-a83f-46ae-9b4a-41878d8f004c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:40.438858 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a6c9498-a83f-46ae-9b4a-41878d8f004c admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:40.450258 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d69c60c6-e690-42bd-bca0-07e591171ccf None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:40.450732 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1374/2772] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Sep 25 04:48:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:48:40.454348 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f931380f-c332-4862-baca-415225772342 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:40.455052 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f931380f-c332-4862-baca-415225772342 service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:40.455335 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f931380f-c332-4862-baca-415225772342 service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:40.455609 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f931380f-c332-4862-baca-415225772342 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:40.455717 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a6c9498-a83f-46ae-9b4a-41878d8f004c admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:40.465099 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f931380f-c332-4862-baca-415225772342 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:40.465702 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a6c9498-a83f-46ae-9b4a-41878d8f004c admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:40.466530 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f931380f-c332-4862-baca-415225772342 service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:40.470805 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daf0ed9b-5e50-4da4-a91d-975b6d365c2f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:40.471735 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daf0ed9b-5e50-4da4-a91d-975b6d365c2f admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:40.472613 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daf0ed9b-5e50-4da4-a91d-975b6d365c2f admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:40.472928 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daf0ed9b-5e50-4da4-a91d-975b6d365c2f admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:40.473567 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daf0ed9b-5e50-4da4-a91d-975b6d365c2f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:40.475463 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daf0ed9b-5e50-4da4-a91d-975b6d365c2f admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:40.477239 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1399/2773] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:48:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 25 04:48:40.505673 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1375/2774] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:48:40 2022] POST /identity/v3/projects => generated 387 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 04:48:40.509555 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f931380f-c332-4862-baca-415225772342 service neutron] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:40.520124 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f931380f-c332-4862-baca-415225772342 service neutron] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:40.579357 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e0d3ec3-dd17-4981-96a2-c001d8ed7df1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:40.579992 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e0d3ec3-dd17-4981-96a2-c001d8ed7df1 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:40.580241 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e0d3ec3-dd17-4981-96a2-c001d8ed7df1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:40.580510 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e0d3ec3-dd17-4981-96a2-c001d8ed7df1 admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:40.581180 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e0d3ec3-dd17-4981-96a2-c001d8ed7df1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:40.583068 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e0d3ec3-dd17-4981-96a2-c001d8ed7df1 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:40.606481 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1400/2775] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:48:40 2022] POST /identity/v3/users => generated 294 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 04:48:40.611421 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daf0ed9b-5e50-4da4-a91d-975b6d365c2f admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:40.622723 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daf0ed9b-5e50-4da4-a91d-975b6d365c2f admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:40.629353 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7242be4f-5a0e-4279-a708-ae33c263c2cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:40.630328 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7242be4f-5a0e-4279-a708-ae33c263c2cc admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:40.630832 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7242be4f-5a0e-4279-a708-ae33c263c2cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:40.631286 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7242be4f-5a0e-4279-a708-ae33c263c2cc admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:40.632194 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7242be4f-5a0e-4279-a708-ae33c263c2cc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:40.633912 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7242be4f-5a0e-4279-a708-ae33c263c2cc admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:40.644996 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1376/2776] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:48:40 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 25 04:48:40.648786 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e0d3ec3-dd17-4981-96a2-c001d8ed7df1 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:40.659401 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e0d3ec3-dd17-4981-96a2-c001d8ed7df1 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:40.664233 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0605e01b-408b-44a8-b7c2-d811908349cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:40.664882 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0605e01b-408b-44a8-b7c2-d811908349cb admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:40.665163 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0605e01b-408b-44a8-b7c2-d811908349cb admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:40.665435 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0605e01b-408b-44a8-b7c2-d811908349cb admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:40.666102 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0605e01b-408b-44a8-b7c2-d811908349cb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:40.667699 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0605e01b-408b-44a8-b7c2-d811908349cb admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:40.678580 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1401/2777] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:48:40 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:48:40.682735 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7242be4f-5a0e-4279-a708-ae33c263c2cc admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:40.693194 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7242be4f-5a0e-4279-a708-ae33c263c2cc admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:40.697913 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5df615e9-0c0e-4056-9345-2c1540734fcc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:40.698643 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df615e9-0c0e-4056-9345-2c1540734fcc admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:40.698926 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df615e9-0c0e-4056-9345-2c1540734fcc admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:40.699204 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df615e9-0c0e-4056-9345-2c1540734fcc admin admin] PATH_INFO: `/v3/projects/add51075f53841f5a7fb1f555f3171c2/users/335276876b274908978d6face5dfa079/roles/8f6cc574056c4278bc9778f14dc4b411` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:40.714329 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5df615e9-0c0e-4056-9345-2c1540734fcc admin admin] RBAC: Authorizing `identity:create_grant(project_id=add51075f53841f5a7fb1f555f3171c2, user_id=335276876b274908978d6face5dfa079, role_id=8f6cc574056c4278bc9778f14dc4b411)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:40.717027 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5df615e9-0c0e-4056-9345-2c1540734fcc admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:40.732315 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1377/2778] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:48:40 2022] PUT /identity/v3/projects/add51075f53841f5a7fb1f555f3171c2/users/335276876b274908978d6face5dfa079/roles/8f6cc574056c4278bc9778f14dc4b411 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:48:40.744442 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d46a8bb-b9a9-4ed1-b4f8-54fd9cced393 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:40.744757 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d46a8bb-b9a9-4ed1-b4f8-54fd9cced393 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:40.745142 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d46a8bb-b9a9-4ed1-b4f8-54fd9cced393 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:40.753481 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5df615e9-0c0e-4056-9345-2c1540734fcc admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:40.757022 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-5df615e9-0c0e-4056-9345-2c1540734fcc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:48:40.772990 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:48:40.778588 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-0d46a8bb-b9a9-4ed1-b4f8-54fd9cced393 None None] MFA Rules not processed for user `335276876b274908978d6face5dfa079`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:48:40.788440 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-0d46a8bb-b9a9-4ed1-b4f8-54fd9cced393 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:48:40.833487 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5df615e9-0c0e-4056-9345-2c1540734fcc admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:40.838596 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55101d89-102b-499e-ba04-72531e86d75d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:40.839494 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55101d89-102b-499e-ba04-72531e86d75d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:40.839774 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55101d89-102b-499e-ba04-72531e86d75d admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:40.840058 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55101d89-102b-499e-ba04-72531e86d75d admin admin] PATH_INFO: `/v3/projects/ca4031a419ae4e8a8f82cbadef2b1fd1` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:40.841197 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55101d89-102b-499e-ba04-72531e86d75d admin admin] RBAC: Authorizing `identity:delete_project(project_id=ca4031a419ae4e8a8f82cbadef2b1fd1)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:40.843120 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55101d89-102b-499e-ba04-72531e86d75d admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:40.850232 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:DELETE=3,keystone:SELECT=95,keystone:INSERT=2 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:48:40.857839 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1402/2779] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:48:40 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:48:40.862819 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d46a8bb-b9a9-4ed1-b4f8-54fd9cced393 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:40.874079 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d46a8bb-b9a9-4ed1-b4f8-54fd9cced393 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:40.879385 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dda90c1-d2c8-4e2a-a455-0048fd7270cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:40.880103 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dda90c1-d2c8-4e2a-a455-0048fd7270cb admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:40.880387 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dda90c1-d2c8-4e2a-a455-0048fd7270cb admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:40.880678 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dda90c1-d2c8-4e2a-a455-0048fd7270cb admin admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:40.881308 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dda90c1-d2c8-4e2a-a455-0048fd7270cb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:40.883193 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dda90c1-d2c8-4e2a-a455-0048fd7270cb admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:40.894788 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-55101d89-102b-499e-ba04-72531e86d75d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ca4031a419ae4e8a8f82cbadef2b1fd1', 'request_id': 'req-55101d89-102b-499e-ba04-72531e86d75d'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:40.903018 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-55101d89-102b-499e-ba04-72531e86d75d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ca4031a419ae4e8a8f82cbadef2b1fd1', 'request_id': 'req-55101d89-102b-499e-ba04-72531e86d75d'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:40.909685 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-55101d89-102b-499e-ba04-72531e86d75d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ca4031a419ae4e8a8f82cbadef2b1fd1', 'request_id': 'req-55101d89-102b-499e-ba04-72531e86d75d'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:40.914580 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1403/2780] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:48:40 2022] POST /identity/v3/projects => generated 387 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 04:48:40.915615 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dda90c1-d2c8-4e2a-a455-0048fd7270cb admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:40.923051 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-55101d89-102b-499e-ba04-72531e86d75d admin admin] The token cache is being invalidate because project ca4031a419ae4e8a8f82cbadef2b1fd1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:48:40.923345 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-55101d89-102b-499e-ba04-72531e86d75d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:40.924035 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-55101d89-102b-499e-ba04-72531e86d75d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:40.925558 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1378/2781] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:48:40 2022] DELETE /identity/v3/projects/ca4031a419ae4e8a8f82cbadef2b1fd1 => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 04:48:40.931646 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cca5fc2-dca4-4aa8-96fd-41d6a5a290b8 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:40.931936 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cca5fc2-dca4-4aa8-96fd-41d6a5a290b8 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:40.932211 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cca5fc2-dca4-4aa8-96fd-41d6a5a290b8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:40.952442 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-6cca5fc2-dca4-4aa8-96fd-41d6a5a290b8 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:48:40.954618 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-6cca5fc2-dca4-4aa8-96fd-41d6a5a290b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:48:40.978128 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dda90c1-d2c8-4e2a-a455-0048fd7270cb admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:40.982843 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af5fa561-fa00-459b-9b52-2d9ac52f3a31 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:40.983455 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af5fa561-fa00-459b-9b52-2d9ac52f3a31 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:40.983736 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af5fa561-fa00-459b-9b52-2d9ac52f3a31 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:40.984041 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af5fa561-fa00-459b-9b52-2d9ac52f3a31 admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:40.984699 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af5fa561-fa00-459b-9b52-2d9ac52f3a31 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:40.986725 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af5fa561-fa00-459b-9b52-2d9ac52f3a31 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:41.012224 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1404/2782] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:48:40 2022] POST /identity/v3/users => generated 300 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 04:48:41.017124 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af5fa561-fa00-459b-9b52-2d9ac52f3a31 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:41.018815 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-af5fa561-fa00-459b-9b52-2d9ac52f3a31 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:48:41.019518 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1379/2783] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:48:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:48:41.026907 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14168f05-7dd9-4d46-bad2-7977176727ff None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:41.027203 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14168f05-7dd9-4d46-bad2-7977176727ff None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:41.027501 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14168f05-7dd9-4d46-bad2-7977176727ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:41.032651 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af5fa561-fa00-459b-9b52-2d9ac52f3a31 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:41.037712 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eba3c5c5-fdd0-4907-931e-ce52bf47ef7d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:41.038356 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eba3c5c5-fdd0-4907-931e-ce52bf47ef7d admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:41.038593 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eba3c5c5-fdd0-4907-931e-ce52bf47ef7d admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:41.038867 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eba3c5c5-fdd0-4907-931e-ce52bf47ef7d admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:41.039505 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eba3c5c5-fdd0-4907-931e-ce52bf47ef7d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:41.041163 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eba3c5c5-fdd0-4907-931e-ce52bf47ef7d admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:41.048292 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-14168f05-7dd9-4d46-bad2-7977176727ff None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:48:41.049547 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-14168f05-7dd9-4d46-bad2-7977176727ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:48:41.054885 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1405/2784] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:48:41 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:48:41.055837 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eba3c5c5-fdd0-4907-931e-ce52bf47ef7d admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:41.060366 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1380/2785] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:48:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 04:48:41.064950 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14168f05-7dd9-4d46-bad2-7977176727ff None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:41.069658 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eba3c5c5-fdd0-4907-931e-ce52bf47ef7d admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:41.074224 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14168f05-7dd9-4d46-bad2-7977176727ff None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:41.075091 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b57d71d-17a7-42b8-8e76-ba1c278a8ed9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:41.075803 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b57d71d-17a7-42b8-8e76-ba1c278a8ed9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:41.076089 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b57d71d-17a7-42b8-8e76-ba1c278a8ed9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:41.076363 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b57d71d-17a7-42b8-8e76-ba1c278a8ed9 admin admin] PATH_INFO: `/v3/projects/c7a852bb886047deb2295de6f2d170ad/users/4875871391a4466a875407eae4d1cc9a/roles/c51fe7c5ad804040879acfb54269455f` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:41.080112 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88e66283-35d4-4907-be2f-1f5b160028d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:41.081396 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88e66283-35d4-4907-be2f-1f5b160028d3 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:41.081396 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88e66283-35d4-4907-be2f-1f5b160028d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:41.081396 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88e66283-35d4-4907-be2f-1f5b160028d3 admin admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:41.081913 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88e66283-35d4-4907-be2f-1f5b160028d3 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:48:41.082219 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88e66283-35d4-4907-be2f-1f5b160028d3 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:41.083826 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88e66283-35d4-4907-be2f-1f5b160028d3 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:41.095821 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b57d71d-17a7-42b8-8e76-ba1c278a8ed9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c7a852bb886047deb2295de6f2d170ad, user_id=4875871391a4466a875407eae4d1cc9a, role_id=c51fe7c5ad804040879acfb54269455f)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:41.096665 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1381/2786] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Sep 25 04:48:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 25 04:48:41.098742 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b57d71d-17a7-42b8-8e76-ba1c278a8ed9 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:41.101064 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88e66283-35d4-4907-be2f-1f5b160028d3 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:41.111848 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88e66283-35d4-4907-be2f-1f5b160028d3 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:41.117704 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1406/2787] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:48:41 2022] PUT /identity/v3/projects/c7a852bb886047deb2295de6f2d170ad/users/4875871391a4466a875407eae4d1cc9a/roles/c51fe7c5ad804040879acfb54269455f => generated 0 bytes in 62 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 04:48:41.122469 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b57d71d-17a7-42b8-8e76-ba1c278a8ed9 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:41.165593 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:48:41.177343 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27870762-339c-491b-9658-e0afe33173dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:41.177981 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27870762-339c-491b-9658-e0afe33173dc admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:41.178260 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27870762-339c-491b-9658-e0afe33173dc admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:41.178536 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27870762-339c-491b-9658-e0afe33173dc admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:41.179164 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27870762-339c-491b-9658-e0afe33173dc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:41.181087 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27870762-339c-491b-9658-e0afe33173dc admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:41.190128 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b57d71d-17a7-42b8-8e76-ba1c278a8ed9 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:41.194820 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91efd8ca-5285-4630-b0bf-4902b3111d5e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:41.195417 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91efd8ca-5285-4630-b0bf-4902b3111d5e admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:41.195694 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91efd8ca-5285-4630-b0bf-4902b3111d5e admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:41.195963 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91efd8ca-5285-4630-b0bf-4902b3111d5e admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:41.196593 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91efd8ca-5285-4630-b0bf-4902b3111d5e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:41.198260 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91efd8ca-5285-4630-b0bf-4902b3111d5e admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:41.203878 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:48:41.209541 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1407/2788] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:48:41 2022] GET /identity/v3/roles => generated 1420 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:48:41.213697 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91efd8ca-5285-4630-b0bf-4902b3111d5e admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:41.218552 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1382/2789] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:48:41 2022] POST /identity/v3/projects => generated 373 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 04:48:41.219429 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27870762-339c-491b-9658-e0afe33173dc admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:41.224793 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91efd8ca-5285-4630-b0bf-4902b3111d5e admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:41.285808 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0e88b09-a803-4c9d-9055-1e7894df2a2b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:41.286471 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e88b09-a803-4c9d-9055-1e7894df2a2b admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:41.286747 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e88b09-a803-4c9d-9055-1e7894df2a2b admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:41.287015 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e88b09-a803-4c9d-9055-1e7894df2a2b admin admin] PATH_INFO: `/v3/projects/c7a852bb886047deb2295de6f2d170ad/users/4875871391a4466a875407eae4d1cc9a/roles/c51fe7c5ad804040879acfb54269455f` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:41.288949 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0e88b09-a803-4c9d-9055-1e7894df2a2b admin admin] RBAC: Authorizing `identity:create_grant(project_id=c7a852bb886047deb2295de6f2d170ad, user_id=4875871391a4466a875407eae4d1cc9a, role_id=c51fe7c5ad804040879acfb54269455f)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:41.291571 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0e88b09-a803-4c9d-9055-1e7894df2a2b admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:41.292338 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27870762-339c-491b-9658-e0afe33173dc admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:41.297725 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c636d47-302b-4fc3-8abc-a2cd48c2cffc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:41.298339 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c636d47-302b-4fc3-8abc-a2cd48c2cffc admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:41.298633 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c636d47-302b-4fc3-8abc-a2cd48c2cffc admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:41.298896 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c636d47-302b-4fc3-8abc-a2cd48c2cffc admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:41.299563 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c636d47-302b-4fc3-8abc-a2cd48c2cffc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:41.301507 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c636d47-302b-4fc3-8abc-a2cd48c2cffc admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:41.302239 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1408/2790] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:48:41 2022] PUT /identity/v3/projects/c7a852bb886047deb2295de6f2d170ad/users/4875871391a4466a875407eae4d1cc9a/roles/c51fe7c5ad804040879acfb54269455f => generated 0 bytes in 89 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 04:48:41.313139 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb8c587f-7738-420b-a04b-a67ffbb41654 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:41.313423 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb8c587f-7738-420b-a04b-a67ffbb41654 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:41.313702 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb8c587f-7738-420b-a04b-a67ffbb41654 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:41.327148 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1383/2791] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:48:41 2022] POST /identity/v3/users => generated 287 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 04:48:41.331168 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c636d47-302b-4fc3-8abc-a2cd48c2cffc admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:41.341135 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c636d47-302b-4fc3-8abc-a2cd48c2cffc admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:41.342304 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-fb8c587f-7738-420b-a04b-a67ffbb41654 None None] MFA Rules not processed for user `4875871391a4466a875407eae4d1cc9a`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:48:41.343780 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-fb8c587f-7738-420b-a04b-a67ffbb41654 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:48:41.398495 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5258bda2-b32a-4376-b8ce-dad4b8e5bf1d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:41.399115 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5258bda2-b32a-4376-b8ce-dad4b8e5bf1d admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:41.399398 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5258bda2-b32a-4376-b8ce-dad4b8e5bf1d admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:41.399674 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5258bda2-b32a-4376-b8ce-dad4b8e5bf1d admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:41.400323 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5258bda2-b32a-4376-b8ce-dad4b8e5bf1d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:41.401961 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5258bda2-b32a-4376-b8ce-dad4b8e5bf1d admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:41.411567 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1384/2792] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:48:41 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 25 04:48:41.413550 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1409/2793] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:48:41 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 04:48:41.415812 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5258bda2-b32a-4376-b8ce-dad4b8e5bf1d admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:41.426177 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb8c587f-7738-420b-a04b-a67ffbb41654 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:41.426426 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5258bda2-b32a-4376-b8ce-dad4b8e5bf1d admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:41.427535 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-fb8c587f-7738-420b-a04b-a67ffbb41654 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:48:41.431058 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b368aca2-085d-4f67-ab3e-b98040651a07 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:41.431638 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b368aca2-085d-4f67-ab3e-b98040651a07 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:41.431918 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b368aca2-085d-4f67-ab3e-b98040651a07 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:41.432191 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b368aca2-085d-4f67-ab3e-b98040651a07 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:41.432860 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b368aca2-085d-4f67-ab3e-b98040651a07 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:41.434456 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b368aca2-085d-4f67-ab3e-b98040651a07 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:41.449941 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1385/2794] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:48:41 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:48:41.451172 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b368aca2-085d-4f67-ab3e-b98040651a07 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:41.461297 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b368aca2-085d-4f67-ab3e-b98040651a07 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:41.466412 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b3f8553-99e8-4abe-9f6a-a16ca31ad385 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:41.467070 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b3f8553-99e8-4abe-9f6a-a16ca31ad385 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:41.467357 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b3f8553-99e8-4abe-9f6a-a16ca31ad385 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:41.467642 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b3f8553-99e8-4abe-9f6a-a16ca31ad385 admin admin] PATH_INFO: `/v3/projects/b981336000334d7d8451feaacc75da99/users/673b4f55a335484d927d7966d72690ad/roles/8f6cc574056c4278bc9778f14dc4b411` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:41.486230 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b3f8553-99e8-4abe-9f6a-a16ca31ad385 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b981336000334d7d8451feaacc75da99, user_id=673b4f55a335484d927d7966d72690ad, role_id=8f6cc574056c4278bc9778f14dc4b411)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:41.488593 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b3f8553-99e8-4abe-9f6a-a16ca31ad385 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:41.492089 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb8c587f-7738-420b-a04b-a67ffbb41654 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:41.496310 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2d8e180-fd30-4e48-9cc2-348847bf97c8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ffbc136c77b84e8c87128f7247f4ec03', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:41.497100 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2d8e180-fd30-4e48-9cc2-348847bf97c8 service cinder] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:41.497385 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2d8e180-fd30-4e48-9cc2-348847bf97c8 service cinder] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:41.497667 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2d8e180-fd30-4e48-9cc2-348847bf97c8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:41.503011 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1386/2795] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:48:41 2022] PUT /identity/v3/projects/b981336000334d7d8451feaacc75da99/users/673b4f55a335484d927d7966d72690ad/roles/8f6cc574056c4278bc9778f14dc4b411 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 04:48:41.505722 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:48:41.508562 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a99dedb-9e14-483b-9670-5881f34ee026 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:41.508944 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a99dedb-9e14-483b-9670-5881f34ee026 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:41.509288 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a99dedb-9e14-483b-9670-5881f34ee026 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:41.511714 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2d8e180-fd30-4e48-9cc2-348847bf97c8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:41.513610 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2d8e180-fd30-4e48-9cc2-348847bf97c8 service cinder] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:41.524763 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1410/2796] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:48:41 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 25 04:48:41.529452 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-2a99dedb-9e14-483b-9670-5881f34ee026 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:48:41.533488 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-2a99dedb-9e14-483b-9670-5881f34ee026 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:48:41.585723 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:48:41.593641 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2d8e180-fd30-4e48-9cc2-348847bf97c8 service cinder] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:41.604846 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1387/2797] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:48:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 04:48:41.605036 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2d8e180-fd30-4e48-9cc2-348847bf97c8 service cinder] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:41.609890 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b0b5b93-7049-4c21-bc04-a3e744b9154c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:41.610623 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b0b5b93-7049-4c21-bc04-a3e744b9154c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:41.610974 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b0b5b93-7049-4c21-bc04-a3e744b9154c admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:41.611317 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b0b5b93-7049-4c21-bc04-a3e744b9154c admin admin] PATH_INFO: `/v3/users/335276876b274908978d6face5dfa079` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:41.612560 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b0b5b93-7049-4c21-bc04-a3e744b9154c admin admin] RBAC: Authorizing `identity:delete_user(user_id=335276876b274908978d6face5dfa079)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:41.614607 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b0b5b93-7049-4c21-bc04-a3e744b9154c admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:41.614799 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a99dedb-9e14-483b-9670-5881f34ee026 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:41.616174 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-2a99dedb-9e14-483b-9670-5881f34ee026 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:48:41.663751 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-7b0b5b93-7049-4c21-bc04-a3e744b9154c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '335276876b274908978d6face5dfa079'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:41.671775 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-7b0b5b93-7049-4c21-bc04-a3e744b9154c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '335276876b274908978d6face5dfa079'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:41.677303 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a99dedb-9e14-483b-9670-5881f34ee026 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:41.680964 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcdfdba6-be12-4229-ae00-583284e2cc51 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:41.681568 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcdfdba6-be12-4229-ae00-583284e2cc51 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:41.681849 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcdfdba6-be12-4229-ae00-583284e2cc51 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:41.682153 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcdfdba6-be12-4229-ae00-583284e2cc51 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:41.683969 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-7b0b5b93-7049-4c21-bc04-a3e744b9154c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '335276876b274908978d6face5dfa079'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:41.684665 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-7b0b5b93-7049-4c21-bc04-a3e744b9154c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '335276876b274908978d6face5dfa079'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:41.692276 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcdfdba6-be12-4229-ae00-583284e2cc51 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:41.692432 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-7b0b5b93-7049-4c21-bc04-a3e744b9154c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '335276876b274908978d6face5dfa079'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:41.693749 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcdfdba6-be12-4229-ae00-583284e2cc51 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:41.698325 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1411/2798] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:48:41 2022] DELETE /identity/v3/users/335276876b274908978d6face5dfa079 => generated 0 bytes in 106 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 04:48:41.703256 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fc86b8a-eddb-4aa0-8d79-6981a8ecc4ce None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:41.703588 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fc86b8a-eddb-4aa0-8d79-6981a8ecc4ce None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:41.703886 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fc86b8a-eddb-4aa0-8d79-6981a8ecc4ce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:41.707335 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1388/2799] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:48:41 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 25 04:48:41.724607 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-2fc86b8a-eddb-4aa0-8d79-6981a8ecc4ce None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:48:41.725925 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-2fc86b8a-eddb-4aa0-8d79-6981a8ecc4ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:48:41.789036 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1412/2800] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:48:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:48:41.799562 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcdfdba6-be12-4229-ae00-583284e2cc51 service neutron] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:41.801445 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-fcdfdba6-be12-4229-ae00-583284e2cc51 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:48:41.823379 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcdfdba6-be12-4229-ae00-583284e2cc51 service neutron] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:41.886998 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea048948-5f36-41c3-9090-5d83ff3f4b43 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:41.887624 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea048948-5f36-41c3-9090-5d83ff3f4b43 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:41.888029 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea048948-5f36-41c3-9090-5d83ff3f4b43 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:41.888319 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea048948-5f36-41c3-9090-5d83ff3f4b43 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:41.898839 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea048948-5f36-41c3-9090-5d83ff3f4b43 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:41.900326 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea048948-5f36-41c3-9090-5d83ff3f4b43 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:41.910549 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1389/2801] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:48:41 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 25 04:48:42.220604 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fc86b8a-eddb-4aa0-8d79-6981a8ecc4ce None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:42.222173 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-2fc86b8a-eddb-4aa0-8d79-6981a8ecc4ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:48:42.237835 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fc86b8a-eddb-4aa0-8d79-6981a8ecc4ce None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:42.244533 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a58395b-8ec2-4f1f-b90e-5d6521d300ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:42.246507 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a58395b-8ec2-4f1f-b90e-5d6521d300ac admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:42.246770 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a58395b-8ec2-4f1f-b90e-5d6521d300ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:42.247051 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a58395b-8ec2-4f1f-b90e-5d6521d300ac admin admin] PATH_INFO: `/v3/projects/add51075f53841f5a7fb1f555f3171c2` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:42.248992 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a58395b-8ec2-4f1f-b90e-5d6521d300ac admin admin] RBAC: Authorizing `identity:delete_project(project_id=add51075f53841f5a7fb1f555f3171c2)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:42.251280 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a58395b-8ec2-4f1f-b90e-5d6521d300ac admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:42.300833 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-5a58395b-8ec2-4f1f-b90e-5d6521d300ac admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'add51075f53841f5a7fb1f555f3171c2', 'request_id': 'req-5a58395b-8ec2-4f1f-b90e-5d6521d300ac'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:42.308772 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-5a58395b-8ec2-4f1f-b90e-5d6521d300ac admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'add51075f53841f5a7fb1f555f3171c2', 'request_id': 'req-5a58395b-8ec2-4f1f-b90e-5d6521d300ac'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:42.313807 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=82,keystone:DELETE=16,keystone:INSERT=2 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:48:42.315993 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-5a58395b-8ec2-4f1f-b90e-5d6521d300ac admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'add51075f53841f5a7fb1f555f3171c2', 'request_id': 'req-5a58395b-8ec2-4f1f-b90e-5d6521d300ac'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:42.331745 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-5a58395b-8ec2-4f1f-b90e-5d6521d300ac admin admin] The token cache is being invalidate because project add51075f53841f5a7fb1f555f3171c2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81434) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:48:42.332042 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-5a58395b-8ec2-4f1f-b90e-5d6521d300ac admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:42.332727 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-5a58395b-8ec2-4f1f-b90e-5d6521d300ac admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:42.334357 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1413/2802] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:48:42 2022] DELETE /identity/v3/projects/add51075f53841f5a7fb1f555f3171c2 => generated 0 bytes in 114 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:48:42.338864 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea048948-5f36-41c3-9090-5d83ff3f4b43 service neutron] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:42.349728 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea048948-5f36-41c3-9090-5d83ff3f4b43 service neutron] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:42.412337 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04a8fbc6-cddb-4df4-9f4f-0803c949b77a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:42.413143 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04a8fbc6-cddb-4df4-9f4f-0803c949b77a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:42.413531 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04a8fbc6-cddb-4df4-9f4f-0803c949b77a admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:42.413905 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04a8fbc6-cddb-4df4-9f4f-0803c949b77a admin admin] PATH_INFO: `/v3/users/4875871391a4466a875407eae4d1cc9a` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:42.415154 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04a8fbc6-cddb-4df4-9f4f-0803c949b77a admin admin] RBAC: Authorizing `identity:delete_user(user_id=4875871391a4466a875407eae4d1cc9a)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:42.418374 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04a8fbc6-cddb-4df4-9f4f-0803c949b77a admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:42.472428 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:48:42.483394 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-04a8fbc6-cddb-4df4-9f4f-0803c949b77a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4875871391a4466a875407eae4d1cc9a'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:42.491857 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-04a8fbc6-cddb-4df4-9f4f-0803c949b77a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4875871391a4466a875407eae4d1cc9a'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:42.508489 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-04a8fbc6-cddb-4df4-9f4f-0803c949b77a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4875871391a4466a875407eae4d1cc9a'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:42.509278 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-04a8fbc6-cddb-4df4-9f4f-0803c949b77a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4875871391a4466a875407eae4d1cc9a'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:42.518353 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-04a8fbc6-cddb-4df4-9f4f-0803c949b77a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4875871391a4466a875407eae4d1cc9a'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:42.525307 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1390/2803] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:48:42 2022] DELETE /identity/v3/users/4875871391a4466a875407eae4d1cc9a => generated 0 bytes in 187 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:48:42.835640 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a58395b-8ec2-4f1f-b90e-5d6521d300ac admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:42.838139 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-5a58395b-8ec2-4f1f-b90e-5d6521d300ac admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:48:42.861191 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a58395b-8ec2-4f1f-b90e-5d6521d300ac admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:42.939949 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2718de93-cff1-476b-95da-98c6d6338a06 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:42.941008 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2718de93-cff1-476b-95da-98c6d6338a06 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:42.941449 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2718de93-cff1-476b-95da-98c6d6338a06 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:42.941879 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2718de93-cff1-476b-95da-98c6d6338a06 admin admin] PATH_INFO: `/v3/projects/c7a852bb886047deb2295de6f2d170ad` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:42.943265 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2718de93-cff1-476b-95da-98c6d6338a06 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c7a852bb886047deb2295de6f2d170ad)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:42.945855 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2718de93-cff1-476b-95da-98c6d6338a06 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:43.007653 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-2718de93-cff1-476b-95da-98c6d6338a06 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c7a852bb886047deb2295de6f2d170ad', 'request_id': 'req-2718de93-cff1-476b-95da-98c6d6338a06'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:43.015810 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-2718de93-cff1-476b-95da-98c6d6338a06 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c7a852bb886047deb2295de6f2d170ad', 'request_id': 'req-2718de93-cff1-476b-95da-98c6d6338a06'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:43.022237 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-2718de93-cff1-476b-95da-98c6d6338a06 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c7a852bb886047deb2295de6f2d170ad', 'request_id': 'req-2718de93-cff1-476b-95da-98c6d6338a06'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:43.033553 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-2718de93-cff1-476b-95da-98c6d6338a06 admin admin] The token cache is being invalidate because project c7a852bb886047deb2295de6f2d170ad was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81434) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:48:43.033862 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-2718de93-cff1-476b-95da-98c6d6338a06 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:43.034513 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-2718de93-cff1-476b-95da-98c6d6338a06 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:43.036297 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1414/2804] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:48:42 2022] DELETE /identity/v3/projects/c7a852bb886047deb2295de6f2d170ad => generated 0 bytes in 201 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:48:43.040943 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2728ea47-bdac-4ba1-a8fc-9e662b6126ba None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:43.041297 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2728ea47-bdac-4ba1-a8fc-9e662b6126ba None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:43.041491 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2728ea47-bdac-4ba1-a8fc-9e662b6126ba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:43.062915 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-2728ea47-bdac-4ba1-a8fc-9e662b6126ba None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:48:43.064159 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-2728ea47-bdac-4ba1-a8fc-9e662b6126ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:48:43.131195 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1391/2805] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:48:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:48:43.139896 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-297f1df0-fe96-4d1d-a6b9-a56cdf3a7049 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:43.141057 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-297f1df0-fe96-4d1d-a6b9-a56cdf3a7049 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:43.141465 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-297f1df0-fe96-4d1d-a6b9-a56cdf3a7049 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:43.165526 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-297f1df0-fe96-4d1d-a6b9-a56cdf3a7049 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:48:43.166757 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-297f1df0-fe96-4d1d-a6b9-a56cdf3a7049 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:48:43.184300 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1415/2806] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:48:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 45 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:48:43.186489 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2728ea47-bdac-4ba1-a8fc-9e662b6126ba None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:43.207562 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2728ea47-bdac-4ba1-a8fc-9e662b6126ba None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:43.213726 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b7fe00b-2ec0-4234-b38c-062fd700116d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:43.214701 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b7fe00b-2ec0-4234-b38c-062fd700116d admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:43.214880 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b7fe00b-2ec0-4234-b38c-062fd700116d admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:43.215151 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b7fe00b-2ec0-4234-b38c-062fd700116d admin admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:43.215951 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b7fe00b-2ec0-4234-b38c-062fd700116d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:48:43.216265 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b7fe00b-2ec0-4234-b38c-062fd700116d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:43.217931 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b7fe00b-2ec0-4234-b38c-062fd700116d admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:43.232444 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1392/2807] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Sep 25 04:48:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:48:43.237661 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-297f1df0-fe96-4d1d-a6b9-a56cdf3a7049 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:43.247297 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-297f1df0-fe96-4d1d-a6b9-a56cdf3a7049 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:43.253898 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e00c5e1-d472-436f-bbca-a7d92ad1c046 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:43.254528 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e00c5e1-d472-436f-bbca-a7d92ad1c046 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:43.254777 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e00c5e1-d472-436f-bbca-a7d92ad1c046 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:43.255047 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e00c5e1-d472-436f-bbca-a7d92ad1c046 admin admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:43.255675 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e00c5e1-d472-436f-bbca-a7d92ad1c046 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:43.257806 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e00c5e1-d472-436f-bbca-a7d92ad1c046 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:43.292672 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1416/2808] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:48:43 2022] POST /identity/v3/projects => generated 387 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:48:43.295611 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b7fe00b-2ec0-4234-b38c-062fd700116d admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:43.306931 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b7fe00b-2ec0-4234-b38c-062fd700116d admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:43.408930 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3068e796-4edd-4544-be47-eca958b4a7eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:43.409670 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3068e796-4edd-4544-be47-eca958b4a7eb admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:43.410065 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3068e796-4edd-4544-be47-eca958b4a7eb admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:43.410419 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3068e796-4edd-4544-be47-eca958b4a7eb admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:43.411080 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3068e796-4edd-4544-be47-eca958b4a7eb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:43.413096 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3068e796-4edd-4544-be47-eca958b4a7eb admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:43.452231 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1393/2809] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:48:43 2022] POST /identity/v3/users => generated 294 bytes in 158 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:48:43.457110 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e00c5e1-d472-436f-bbca-a7d92ad1c046 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:43.475069 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e00c5e1-d472-436f-bbca-a7d92ad1c046 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:43.480622 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d556e26-9835-460f-ac8c-189b980c48ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:43.481429 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d556e26-9835-460f-ac8c-189b980c48ee admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:43.481706 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d556e26-9835-460f-ac8c-189b980c48ee admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:43.481978 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d556e26-9835-460f-ac8c-189b980c48ee admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:43.482630 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d556e26-9835-460f-ac8c-189b980c48ee admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:43.484545 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d556e26-9835-460f-ac8c-189b980c48ee admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:43.496365 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1417/2810] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:48:43 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:48:43.500528 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3068e796-4edd-4544-be47-eca958b4a7eb admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:43.520813 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3068e796-4edd-4544-be47-eca958b4a7eb admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:43.525773 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec5781b3-6c82-4981-b3f9-f391f772b5bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:43.526393 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec5781b3-6c82-4981-b3f9-f391f772b5bb admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:43.526679 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec5781b3-6c82-4981-b3f9-f391f772b5bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:43.526951 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec5781b3-6c82-4981-b3f9-f391f772b5bb admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:43.527596 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec5781b3-6c82-4981-b3f9-f391f772b5bb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:43.529307 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec5781b3-6c82-4981-b3f9-f391f772b5bb admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:43.541721 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1394/2811] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:48:43 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:48:43.545885 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d556e26-9835-460f-ac8c-189b980c48ee admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:43.555811 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d556e26-9835-460f-ac8c-189b980c48ee admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:43.560476 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d39a717-0eb7-4b8c-864a-5c2a062419db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:43.561164 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d39a717-0eb7-4b8c-864a-5c2a062419db admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:43.561442 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d39a717-0eb7-4b8c-864a-5c2a062419db admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:43.561708 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d39a717-0eb7-4b8c-864a-5c2a062419db admin admin] PATH_INFO: `/v3/projects/30c094f160764afcae899242d423ff11/users/19cf98c3b49744c88063cc47b21b40f0/roles/8f6cc574056c4278bc9778f14dc4b411` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:43.578681 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d39a717-0eb7-4b8c-864a-5c2a062419db admin admin] RBAC: Authorizing `identity:create_grant(project_id=30c094f160764afcae899242d423ff11, user_id=19cf98c3b49744c88063cc47b21b40f0, role_id=8f6cc574056c4278bc9778f14dc4b411)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:43.581342 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d39a717-0eb7-4b8c-864a-5c2a062419db admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:43.595918 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1418/2812] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:48:43 2022] PUT /identity/v3/projects/30c094f160764afcae899242d423ff11/users/19cf98c3b49744c88063cc47b21b40f0/roles/8f6cc574056c4278bc9778f14dc4b411 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 04:48:43.606264 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2f66c8a-ed32-482b-8c89-b1a314fd9b3a None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:43.606487 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2f66c8a-ed32-482b-8c89-b1a314fd9b3a None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:43.606762 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2f66c8a-ed32-482b-8c89-b1a314fd9b3a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:43.636322 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-b2f66c8a-ed32-482b-8c89-b1a314fd9b3a None None] MFA Rules not processed for user `19cf98c3b49744c88063cc47b21b40f0`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:48:43.637607 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-b2f66c8a-ed32-482b-8c89-b1a314fd9b3a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:48:43.647783 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:DELETE=5,keystone:SELECT=91,keystone:INSERT=4 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:48:43.709165 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1395/2813] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:48:43 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 04:48:43.713822 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d39a717-0eb7-4b8c-864a-5c2a062419db admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:43.732097 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:DELETE=8,keystone:SELECT=89,keystone:INSERT=3 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:48:43.781481 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d39a717-0eb7-4b8c-864a-5c2a062419db admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:43.786469 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31ea6130-8ac7-4dc2-9346-a55ae2911df3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:43.787082 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31ea6130-8ac7-4dc2-9346-a55ae2911df3 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:43.787367 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31ea6130-8ac7-4dc2-9346-a55ae2911df3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:43.787691 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31ea6130-8ac7-4dc2-9346-a55ae2911df3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:43.788351 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31ea6130-8ac7-4dc2-9346-a55ae2911df3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:43.790376 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31ea6130-8ac7-4dc2-9346-a55ae2911df3 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:43.823756 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1419/2814] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:48:43 2022] POST /identity/v3/projects => generated 387 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:48:43.828109 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2f66c8a-ed32-482b-8c89-b1a314fd9b3a None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:43.841699 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2f66c8a-ed32-482b-8c89-b1a314fd9b3a None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:43.907189 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bd4fbf3-21c5-42ee-a653-7e6b92b19066 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:43.907814 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bd4fbf3-21c5-42ee-a653-7e6b92b19066 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:43.908095 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bd4fbf3-21c5-42ee-a653-7e6b92b19066 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:43.908368 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bd4fbf3-21c5-42ee-a653-7e6b92b19066 admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:43.909051 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bd4fbf3-21c5-42ee-a653-7e6b92b19066 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:43.911037 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bd4fbf3-21c5-42ee-a653-7e6b92b19066 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:43.939088 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1396/2815] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:48:43 2022] POST /identity/v3/users => generated 300 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:48:43.943195 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31ea6130-8ac7-4dc2-9346-a55ae2911df3 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:43.954770 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31ea6130-8ac7-4dc2-9346-a55ae2911df3 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:43.961983 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca65a633-3041-419a-8c87-fd4f9f24c0cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:43.962746 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca65a633-3041-419a-8c87-fd4f9f24c0cc admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:43.963074 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca65a633-3041-419a-8c87-fd4f9f24c0cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:43.963350 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca65a633-3041-419a-8c87-fd4f9f24c0cc admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:43.964377 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca65a633-3041-419a-8c87-fd4f9f24c0cc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:43.972767 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca65a633-3041-419a-8c87-fd4f9f24c0cc admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:43.989731 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1420/2816] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:48:43 2022] GET /identity/v3/roles => generated 1420 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:48:43.990793 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bd4fbf3-21c5-42ee-a653-7e6b92b19066 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:44.003030 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bd4fbf3-21c5-42ee-a653-7e6b92b19066 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:44.008791 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-049eb08c-cccf-422a-a0e2-6608bf8efa94 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:44.009672 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-049eb08c-cccf-422a-a0e2-6608bf8efa94 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:44.010225 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-049eb08c-cccf-422a-a0e2-6608bf8efa94 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:44.010613 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-049eb08c-cccf-422a-a0e2-6608bf8efa94 admin admin] PATH_INFO: `/v3/projects/f68ddb80a5d4464483e60d7bccbb2845/users/b16e72477fa04068b9e1d2a0f0a552b8/roles/c51fe7c5ad804040879acfb54269455f` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:44.035087 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-049eb08c-cccf-422a-a0e2-6608bf8efa94 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f68ddb80a5d4464483e60d7bccbb2845, user_id=b16e72477fa04068b9e1d2a0f0a552b8, role_id=c51fe7c5ad804040879acfb54269455f)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:44.039521 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-049eb08c-cccf-422a-a0e2-6608bf8efa94 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:44.064771 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca65a633-3041-419a-8c87-fd4f9f24c0cc admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:44.065884 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1397/2817] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:48:43 2022] PUT /identity/v3/projects/f68ddb80a5d4464483e60d7bccbb2845/users/b16e72477fa04068b9e1d2a0f0a552b8/roles/c51fe7c5ad804040879acfb54269455f => generated 0 bytes in 70 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:48:44.079005 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca65a633-3041-419a-8c87-fd4f9f24c0cc admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:44.160822 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-061870a3-6a38-4d4f-8421-fde353992bbe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:44.161456 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-061870a3-6a38-4d4f-8421-fde353992bbe admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:44.161791 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-061870a3-6a38-4d4f-8421-fde353992bbe admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:44.162072 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-061870a3-6a38-4d4f-8421-fde353992bbe admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:44.162730 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-061870a3-6a38-4d4f-8421-fde353992bbe admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:44.164363 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-061870a3-6a38-4d4f-8421-fde353992bbe admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:44.177587 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1421/2818] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:48:44 2022] GET /identity/v3/roles => generated 1420 bytes in 113 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:48:44.181865 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-049eb08c-cccf-422a-a0e2-6608bf8efa94 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:44.194209 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-049eb08c-cccf-422a-a0e2-6608bf8efa94 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:44.207652 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89e466ac-e640-4e10-864e-48b36dc3abd8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:44.208362 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89e466ac-e640-4e10-864e-48b36dc3abd8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:44.208672 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89e466ac-e640-4e10-864e-48b36dc3abd8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:44.209111 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89e466ac-e640-4e10-864e-48b36dc3abd8 admin admin] PATH_INFO: `/v3/projects/f68ddb80a5d4464483e60d7bccbb2845/users/b16e72477fa04068b9e1d2a0f0a552b8/roles/c51fe7c5ad804040879acfb54269455f` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:44.215677 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89e466ac-e640-4e10-864e-48b36dc3abd8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f68ddb80a5d4464483e60d7bccbb2845, user_id=b16e72477fa04068b9e1d2a0f0a552b8, role_id=c51fe7c5ad804040879acfb54269455f)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:44.219651 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89e466ac-e640-4e10-864e-48b36dc3abd8 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:44.236983 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1398/2819] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:48:44 2022] PUT /identity/v3/projects/f68ddb80a5d4464483e60d7bccbb2845/users/b16e72477fa04068b9e1d2a0f0a552b8/roles/c51fe7c5ad804040879acfb54269455f => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:48:44.250139 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c89fc4b-7e2a-476a-b970-f30a5fb06df2 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:44.250932 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c89fc4b-7e2a-476a-b970-f30a5fb06df2 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:44.251065 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c89fc4b-7e2a-476a-b970-f30a5fb06df2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:44.284576 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-7c89fc4b-7e2a-476a-b970-f30a5fb06df2 None None] MFA Rules not processed for user `b16e72477fa04068b9e1d2a0f0a552b8`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:48:44.286752 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-7c89fc4b-7e2a-476a-b970-f30a5fb06df2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:48:44.376398 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1422/2820] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:48:44 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:48:44.383459 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89e466ac-e640-4e10-864e-48b36dc3abd8 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:44.385725 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-89e466ac-e640-4e10-864e-48b36dc3abd8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:48:44.489570 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89e466ac-e640-4e10-864e-48b36dc3abd8 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:44.496820 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dc7ef1b-5bdf-4f37-82d3-41880ec00c75 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ffbc136c77b84e8c87128f7247f4ec03', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:44.497711 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dc7ef1b-5bdf-4f37-82d3-41880ec00c75 service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:44.498540 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dc7ef1b-5bdf-4f37-82d3-41880ec00c75 service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:44.498983 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dc7ef1b-5bdf-4f37-82d3-41880ec00c75 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:44.511360 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dc7ef1b-5bdf-4f37-82d3-41880ec00c75 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:44.514127 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dc7ef1b-5bdf-4f37-82d3-41880ec00c75 service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:44.527448 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1399/2821] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:48:44 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 144 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 25 04:48:44.723118 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c89fc4b-7e2a-476a-b970-f30a5fb06df2 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:44.736524 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:48:44.795749 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c89fc4b-7e2a-476a-b970-f30a5fb06df2 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:44.801255 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8187837a-3cd6-4b08-8043-8803d0a27b06 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:44.801951 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8187837a-3cd6-4b08-8043-8803d0a27b06 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:44.802442 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8187837a-3cd6-4b08-8043-8803d0a27b06 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:44.802724 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8187837a-3cd6-4b08-8043-8803d0a27b06 admin admin] PATH_INFO: `/v3/users/19cf98c3b49744c88063cc47b21b40f0` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:44.803881 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8187837a-3cd6-4b08-8043-8803d0a27b06 admin admin] RBAC: Authorizing `identity:delete_user(user_id=19cf98c3b49744c88063cc47b21b40f0)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:44.806007 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8187837a-3cd6-4b08-8043-8803d0a27b06 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:44.869706 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-8187837a-3cd6-4b08-8043-8803d0a27b06 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '19cf98c3b49744c88063cc47b21b40f0'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:44.880941 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-8187837a-3cd6-4b08-8043-8803d0a27b06 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '19cf98c3b49744c88063cc47b21b40f0'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:44.899120 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-8187837a-3cd6-4b08-8043-8803d0a27b06 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '19cf98c3b49744c88063cc47b21b40f0'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:44.900499 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-8187837a-3cd6-4b08-8043-8803d0a27b06 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '19cf98c3b49744c88063cc47b21b40f0'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:44.910671 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-8187837a-3cd6-4b08-8043-8803d0a27b06 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '19cf98c3b49744c88063cc47b21b40f0'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:44.918152 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1423/2822] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:48:44 2022] DELETE /identity/v3/users/19cf98c3b49744c88063cc47b21b40f0 => generated 0 bytes in 195 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 04:48:44.923301 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b7bbcad-8c4c-4c63-9350-076a7122c8e9 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:44.923662 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b7bbcad-8c4c-4c63-9350-076a7122c8e9 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:44.923985 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b7bbcad-8c4c-4c63-9350-076a7122c8e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:44.938512 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:48:44.946193 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-7b7bbcad-8c4c-4c63-9350-076a7122c8e9 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:48:44.948692 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-7b7bbcad-8c4c-4c63-9350-076a7122c8e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:48:45.056728 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1400/2823] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:48:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:48:45.067377 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8187837a-3cd6-4b08-8043-8803d0a27b06 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:45.068973 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-8187837a-3cd6-4b08-8043-8803d0a27b06 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:48:45.167112 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8187837a-3cd6-4b08-8043-8803d0a27b06 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:45.173971 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ca5fb83-c3f3-40fc-8077-c264fc2f9935 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:45.175143 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca5fb83-c3f3-40fc-8077-c264fc2f9935 service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:45.175505 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca5fb83-c3f3-40fc-8077-c264fc2f9935 service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:45.175850 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca5fb83-c3f3-40fc-8077-c264fc2f9935 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:45.189205 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ca5fb83-c3f3-40fc-8077-c264fc2f9935 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:45.191493 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ca5fb83-c3f3-40fc-8077-c264fc2f9935 service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:45.205363 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1424/2824] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:48:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 139 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:48:45.710410 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b7bbcad-8c4c-4c63-9350-076a7122c8e9 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:45.711401 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-7b7bbcad-8c4c-4c63-9350-076a7122c8e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:48:45.725942 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b7bbcad-8c4c-4c63-9350-076a7122c8e9 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:45.732300 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35dc83fb-0ead-4bb9-9d51-8fd6dd995cb0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:45.733114 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35dc83fb-0ead-4bb9-9d51-8fd6dd995cb0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:45.733309 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35dc83fb-0ead-4bb9-9d51-8fd6dd995cb0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:45.733732 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35dc83fb-0ead-4bb9-9d51-8fd6dd995cb0 admin admin] PATH_INFO: `/v3/projects/30c094f160764afcae899242d423ff11` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:45.734893 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35dc83fb-0ead-4bb9-9d51-8fd6dd995cb0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=30c094f160764afcae899242d423ff11)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:45.738068 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35dc83fb-0ead-4bb9-9d51-8fd6dd995cb0 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:45.853674 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-35dc83fb-0ead-4bb9-9d51-8fd6dd995cb0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '30c094f160764afcae899242d423ff11', 'request_id': 'req-35dc83fb-0ead-4bb9-9d51-8fd6dd995cb0'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:45.864209 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-35dc83fb-0ead-4bb9-9d51-8fd6dd995cb0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '30c094f160764afcae899242d423ff11', 'request_id': 'req-35dc83fb-0ead-4bb9-9d51-8fd6dd995cb0'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:45.872221 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-35dc83fb-0ead-4bb9-9d51-8fd6dd995cb0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '30c094f160764afcae899242d423ff11', 'request_id': 'req-35dc83fb-0ead-4bb9-9d51-8fd6dd995cb0'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:45.887209 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-35dc83fb-0ead-4bb9-9d51-8fd6dd995cb0 admin admin] The token cache is being invalidate because project 30c094f160764afcae899242d423ff11 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:48:45.887564 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-35dc83fb-0ead-4bb9-9d51-8fd6dd995cb0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:45.888196 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-35dc83fb-0ead-4bb9-9d51-8fd6dd995cb0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:45.891038 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1401/2825] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:48:45 2022] DELETE /identity/v3/projects/30c094f160764afcae899242d423ff11 => generated 0 bytes in 182 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:48:45.897833 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ca5fb83-c3f3-40fc-8077-c264fc2f9935 service neutron] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:45.910002 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ca5fb83-c3f3-40fc-8077-c264fc2f9935 service neutron] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:45.945441 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:48:45.978441 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fc54c9b-7cf3-45c6-8090-5b5d6c55c421 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:45.979098 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fc54c9b-7cf3-45c6-8090-5b5d6c55c421 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:45.979380 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fc54c9b-7cf3-45c6-8090-5b5d6c55c421 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:45.979654 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fc54c9b-7cf3-45c6-8090-5b5d6c55c421 admin admin] PATH_INFO: `/v3/users/b16e72477fa04068b9e1d2a0f0a552b8` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:45.981170 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fc54c9b-7cf3-45c6-8090-5b5d6c55c421 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b16e72477fa04068b9e1d2a0f0a552b8)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:45.984686 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fc54c9b-7cf3-45c6-8090-5b5d6c55c421 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:46.064943 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-6fc54c9b-7cf3-45c6-8090-5b5d6c55c421 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b16e72477fa04068b9e1d2a0f0a552b8'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:46.079385 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-6fc54c9b-7cf3-45c6-8090-5b5d6c55c421 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b16e72477fa04068b9e1d2a0f0a552b8'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:46.101608 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-6fc54c9b-7cf3-45c6-8090-5b5d6c55c421 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b16e72477fa04068b9e1d2a0f0a552b8'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:46.102538 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-6fc54c9b-7cf3-45c6-8090-5b5d6c55c421 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b16e72477fa04068b9e1d2a0f0a552b8'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:46.114482 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-6fc54c9b-7cf3-45c6-8090-5b5d6c55c421 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b16e72477fa04068b9e1d2a0f0a552b8'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:46.134895 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1425/2826] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:48:45 2022] DELETE /identity/v3/users/b16e72477fa04068b9e1d2a0f0a552b8 => generated 0 bytes in 227 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:48:46.265989 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf34269-ce7a-4333-92f2-cd3fa7be5081 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:46.267145 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf34269-ce7a-4333-92f2-cd3fa7be5081 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:46.267437 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf34269-ce7a-4333-92f2-cd3fa7be5081 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:46.300308 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-cbf34269-ce7a-4333-92f2-cd3fa7be5081 None None] MFA Rules not processed for user `673b4f55a335484d927d7966d72690ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:48:46.301853 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-cbf34269-ce7a-4333-92f2-cd3fa7be5081 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:48:46.383973 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1402/2827] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:48:46 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:48:46.388471 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fc54c9b-7cf3-45c6-8090-5b5d6c55c421 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:46.390578 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-6fc54c9b-7cf3-45c6-8090-5b5d6c55c421 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:48:46.450182 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbf34269-ce7a-4333-92f2-cd3fa7be5081 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:46.451692 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-cbf34269-ce7a-4333-92f2-cd3fa7be5081 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:48:46.472732 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fc54c9b-7cf3-45c6-8090-5b5d6c55c421 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:46.478818 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c34804eb-d71a-47bf-8b9d-fa92778e833a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:46.479432 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c34804eb-d71a-47bf-8b9d-fa92778e833a admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:46.479757 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c34804eb-d71a-47bf-8b9d-fa92778e833a admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:46.480055 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c34804eb-d71a-47bf-8b9d-fa92778e833a admin admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:46.480910 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c34804eb-d71a-47bf-8b9d-fa92778e833a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:46.485775 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c34804eb-d71a-47bf-8b9d-fa92778e833a admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:46.506302 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:48:46.519311 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1426/2828] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:48:46 2022] POST /identity/v3/projects => generated 371 bytes in 131 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:48:46.538644 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c34804eb-d71a-47bf-8b9d-fa92778e833a admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:46.572628 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:48:46.574436 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbf34269-ce7a-4333-92f2-cd3fa7be5081 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:46.595898 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c34804eb-d71a-47bf-8b9d-fa92778e833a admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:46.600593 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28f16114-061e-49c0-ae74-903937a5b2e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:46.601229 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28f16114-061e-49c0-ae74-903937a5b2e3 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:46.601507 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28f16114-061e-49c0-ae74-903937a5b2e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:46.601775 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28f16114-061e-49c0-ae74-903937a5b2e3 admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:46.602401 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28f16114-061e-49c0-ae74-903937a5b2e3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:46.604320 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28f16114-061e-49c0-ae74-903937a5b2e3 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:46.629309 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1427/2829] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:48:46 2022] POST /identity/v3/users => generated 292 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:48:46.633610 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28f16114-061e-49c0-ae74-903937a5b2e3 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:46.639710 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54e11a34-f7f9-4933-8522-0f4228e2cfbd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:46.640593 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e11a34-f7f9-4933-8522-0f4228e2cfbd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:46.640593 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e11a34-f7f9-4933-8522-0f4228e2cfbd admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:46.640905 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e11a34-f7f9-4933-8522-0f4228e2cfbd admin admin] PATH_INFO: `/v3/projects/f68ddb80a5d4464483e60d7bccbb2845` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:46.642186 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54e11a34-f7f9-4933-8522-0f4228e2cfbd admin admin] RBAC: Authorizing `identity:delete_project(project_id=f68ddb80a5d4464483e60d7bccbb2845)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:46.644255 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54e11a34-f7f9-4933-8522-0f4228e2cfbd admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:46.644960 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28f16114-061e-49c0-ae74-903937a5b2e3 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:46.650023 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d703cc71-69d5-4fb2-9766-e13da618a945 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:46.650629 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d703cc71-69d5-4fb2-9766-e13da618a945 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:46.651372 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d703cc71-69d5-4fb2-9766-e13da618a945 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:46.651653 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d703cc71-69d5-4fb2-9766-e13da618a945 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:46.652299 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d703cc71-69d5-4fb2-9766-e13da618a945 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:46.654329 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d703cc71-69d5-4fb2-9766-e13da618a945 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:46.665845 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1428/2830] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:48:46 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:48:46.669976 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d703cc71-69d5-4fb2-9766-e13da618a945 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:46.680245 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d703cc71-69d5-4fb2-9766-e13da618a945 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:46.692876 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-54e11a34-f7f9-4933-8522-0f4228e2cfbd admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f68ddb80a5d4464483e60d7bccbb2845', 'request_id': 'req-54e11a34-f7f9-4933-8522-0f4228e2cfbd'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:46.700736 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-54e11a34-f7f9-4933-8522-0f4228e2cfbd admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f68ddb80a5d4464483e60d7bccbb2845', 'request_id': 'req-54e11a34-f7f9-4933-8522-0f4228e2cfbd'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:46.706739 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-54e11a34-f7f9-4933-8522-0f4228e2cfbd admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f68ddb80a5d4464483e60d7bccbb2845', 'request_id': 'req-54e11a34-f7f9-4933-8522-0f4228e2cfbd'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:46.719593 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-54e11a34-f7f9-4933-8522-0f4228e2cfbd admin admin] The token cache is being invalidate because project f68ddb80a5d4464483e60d7bccbb2845 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:48:46.720224 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-54e11a34-f7f9-4933-8522-0f4228e2cfbd admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:46.720666 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-54e11a34-f7f9-4933-8522-0f4228e2cfbd admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:46.722676 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1403/2831] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:48:46 2022] DELETE /identity/v3/projects/f68ddb80a5d4464483e60d7bccbb2845 => generated 0 bytes in 273 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:48:46.727963 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e15c333-7e35-43b6-b51c-62f26d96e92c None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:46.728371 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e15c333-7e35-43b6-b51c-62f26d96e92c None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:46.728747 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e15c333-7e35-43b6-b51c-62f26d96e92c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:46.744747 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1c87d5a-68f5-4076-86c8-15d87e466e99 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:46.745419 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1c87d5a-68f5-4076-86c8-15d87e466e99 admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:46.745702 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1c87d5a-68f5-4076-86c8-15d87e466e99 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:46.745979 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1c87d5a-68f5-4076-86c8-15d87e466e99 admin admin] PATH_INFO: `/v3/projects/df48a331b7d444bc9fee0f035a630852/users/ef85cd347e6546229f92a41b74e3eb5d/roles/c51fe7c5ad804040879acfb54269455f` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:46.749115 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-2e15c333-7e35-43b6-b51c-62f26d96e92c None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:48:46.750673 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-2e15c333-7e35-43b6-b51c-62f26d96e92c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:48:46.759082 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1404/2832] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:48:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 04:48:46.761743 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1c87d5a-68f5-4076-86c8-15d87e466e99 admin admin] RBAC: Authorizing `identity:create_grant(project_id=df48a331b7d444bc9fee0f035a630852, user_id=ef85cd347e6546229f92a41b74e3eb5d, role_id=c51fe7c5ad804040879acfb54269455f)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:46.764374 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1c87d5a-68f5-4076-86c8-15d87e466e99 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:46.767342 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55b8aba5-eb01-46e3-8358-fb3b430cbde1 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:46.767626 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55b8aba5-eb01-46e3-8358-fb3b430cbde1 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:46.767905 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55b8aba5-eb01-46e3-8358-fb3b430cbde1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:46.778094 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1429/2833] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:48:46 2022] PUT /identity/v3/projects/df48a331b7d444bc9fee0f035a630852/users/ef85cd347e6546229f92a41b74e3eb5d/roles/c51fe7c5ad804040879acfb54269455f => generated 0 bytes in 109 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 04:48:46.783258 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1c87d5a-68f5-4076-86c8-15d87e466e99 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:46.784888 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-a1c87d5a-68f5-4076-86c8-15d87e466e99 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:48:46.790465 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-55b8aba5-eb01-46e3-8358-fb3b430cbde1 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:48:46.791958 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-55b8aba5-eb01-46e3-8358-fb3b430cbde1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:48:46.839250 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:48:46.853924 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1c87d5a-68f5-4076-86c8-15d87e466e99 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:46.856256 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1405/2834] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:48:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 04:48:46.858618 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e669aff-5ef5-492b-823b-c9a32216fe63 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:46.859955 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e669aff-5ef5-492b-823b-c9a32216fe63 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:46.860255 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e669aff-5ef5-492b-823b-c9a32216fe63 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:46.860529 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e669aff-5ef5-492b-823b-c9a32216fe63 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:46.861209 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e669aff-5ef5-492b-823b-c9a32216fe63 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:46.861383 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55b8aba5-eb01-46e3-8358-fb3b430cbde1 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:46.862818 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e669aff-5ef5-492b-823b-c9a32216fe63 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:46.870663 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55b8aba5-eb01-46e3-8358-fb3b430cbde1 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:46.873552 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1430/2835] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:48:46 2022] GET /identity/v3/roles => generated 1420 bytes in 91 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 25 04:48:46.877884 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e669aff-5ef5-492b-823b-c9a32216fe63 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:46.878828 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc3d6bc0-0622-4efc-8604-a50b81257b4d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:46.879502 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc3d6bc0-0622-4efc-8604-a50b81257b4d admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:46.879735 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc3d6bc0-0622-4efc-8604-a50b81257b4d admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:46.880042 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc3d6bc0-0622-4efc-8604-a50b81257b4d admin admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:46.880722 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc3d6bc0-0622-4efc-8604-a50b81257b4d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:48:46.881034 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc3d6bc0-0622-4efc-8604-a50b81257b4d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:46.882721 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc3d6bc0-0622-4efc-8604-a50b81257b4d admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:46.887655 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e669aff-5ef5-492b-823b-c9a32216fe63 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:46.892358 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d156590-8823-4535-a0a5-ed53fafb8c59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:46.893211 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d156590-8823-4535-a0a5-ed53fafb8c59 admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:46.893492 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d156590-8823-4535-a0a5-ed53fafb8c59 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:46.893763 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d156590-8823-4535-a0a5-ed53fafb8c59 admin admin] PATH_INFO: `/v3/projects/df48a331b7d444bc9fee0f035a630852/users/ef85cd347e6546229f92a41b74e3eb5d/roles/c51fe7c5ad804040879acfb54269455f` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:46.898792 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1406/2836] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Sep 25 04:48:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:48:46.899099 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d156590-8823-4535-a0a5-ed53fafb8c59 admin admin] RBAC: Authorizing `identity:create_grant(project_id=df48a331b7d444bc9fee0f035a630852, user_id=ef85cd347e6546229f92a41b74e3eb5d, role_id=c51fe7c5ad804040879acfb54269455f)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:46.899099 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d156590-8823-4535-a0a5-ed53fafb8c59 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:46.901264 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc3d6bc0-0622-4efc-8604-a50b81257b4d admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:46.912238 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1431/2837] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:48:46 2022] PUT /identity/v3/projects/df48a331b7d444bc9fee0f035a630852/users/ef85cd347e6546229f92a41b74e3eb5d/roles/c51fe7c5ad804040879acfb54269455f => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 04:48:46.914222 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc3d6bc0-0622-4efc-8604-a50b81257b4d admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:46.982184 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53730d39-00de-4d9d-b46f-7d3807152ee8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:46.982800 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53730d39-00de-4d9d-b46f-7d3807152ee8 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:46.983085 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53730d39-00de-4d9d-b46f-7d3807152ee8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:46.983356 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53730d39-00de-4d9d-b46f-7d3807152ee8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:46.984829 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53730d39-00de-4d9d-b46f-7d3807152ee8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:46.986875 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53730d39-00de-4d9d-b46f-7d3807152ee8 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:47.025240 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1407/2838] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:48:46 2022] POST /identity/v3/projects => generated 385 bytes in 124 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:48:47.030078 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d156590-8823-4535-a0a5-ed53fafb8c59 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:47.044683 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d156590-8823-4535-a0a5-ed53fafb8c59 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:47.069537 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:48:47.143003 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fef136a0-7ed6-4d82-a827-dcade2fa0386 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:47.145038 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fef136a0-7ed6-4d82-a827-dcade2fa0386 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:47.145826 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fef136a0-7ed6-4d82-a827-dcade2fa0386 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:47.146003 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fef136a0-7ed6-4d82-a827-dcade2fa0386 admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:47.146890 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fef136a0-7ed6-4d82-a827-dcade2fa0386 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:47.148744 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fef136a0-7ed6-4d82-a827-dcade2fa0386 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:47.177650 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1432/2839] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:48:47 2022] POST /identity/v3/users => generated 293 bytes in 149 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:48:47.181663 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53730d39-00de-4d9d-b46f-7d3807152ee8 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:47.198826 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53730d39-00de-4d9d-b46f-7d3807152ee8 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:47.205320 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de3cfb5b-3e7c-40d0-b481-86bcb64fb2cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:47.205971 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de3cfb5b-3e7c-40d0-b481-86bcb64fb2cb admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:47.206252 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de3cfb5b-3e7c-40d0-b481-86bcb64fb2cb admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:47.206528 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de3cfb5b-3e7c-40d0-b481-86bcb64fb2cb admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:47.207197 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de3cfb5b-3e7c-40d0-b481-86bcb64fb2cb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:47.208894 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de3cfb5b-3e7c-40d0-b481-86bcb64fb2cb admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:47.226040 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1408/2840] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:48:47 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 25 04:48:47.229684 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fef136a0-7ed6-4d82-a827-dcade2fa0386 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:47.241147 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fef136a0-7ed6-4d82-a827-dcade2fa0386 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:47.252359 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c655d848-2e0f-4fc5-810d-62b76cd9d520 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:47.253653 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c655d848-2e0f-4fc5-810d-62b76cd9d520 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:47.254186 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c655d848-2e0f-4fc5-810d-62b76cd9d520 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:47.254681 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c655d848-2e0f-4fc5-810d-62b76cd9d520 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:47.255594 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c655d848-2e0f-4fc5-810d-62b76cd9d520 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:47.257533 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c655d848-2e0f-4fc5-810d-62b76cd9d520 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:47.275870 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1433/2841] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:48:47 2022] GET /identity/v3/roles => generated 1420 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:48:47.280040 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de3cfb5b-3e7c-40d0-b481-86bcb64fb2cb admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:47.293578 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de3cfb5b-3e7c-40d0-b481-86bcb64fb2cb admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:47.300189 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-516ac638-e139-42dd-b74c-37bfdc025293 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:47.301503 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-516ac638-e139-42dd-b74c-37bfdc025293 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:47.301503 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-516ac638-e139-42dd-b74c-37bfdc025293 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:47.304002 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-516ac638-e139-42dd-b74c-37bfdc025293 admin admin] PATH_INFO: `/v3/projects/926a6ac0c3664ef6840c4ab8873896e0/users/b8ecd0b57d4f4d238eae8a3adb67dfe5/roles/8f6cc574056c4278bc9778f14dc4b411` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:47.326832 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-516ac638-e139-42dd-b74c-37bfdc025293 admin admin] RBAC: Authorizing `identity:create_grant(project_id=926a6ac0c3664ef6840c4ab8873896e0, user_id=b8ecd0b57d4f4d238eae8a3adb67dfe5, role_id=8f6cc574056c4278bc9778f14dc4b411)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:47.330519 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-516ac638-e139-42dd-b74c-37bfdc025293 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:47.365412 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1409/2842] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:48:47 2022] PUT /identity/v3/projects/926a6ac0c3664ef6840c4ab8873896e0/users/b8ecd0b57d4f4d238eae8a3adb67dfe5/roles/8f6cc574056c4278bc9778f14dc4b411 => generated 0 bytes in 86 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:48:47.376537 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f57777-4f6f-4c66-9fbe-bc91e2c34893 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:47.376851 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f57777-4f6f-4c66-9fbe-bc91e2c34893 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:47.377132 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f57777-4f6f-4c66-9fbe-bc91e2c34893 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:47.408329 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-01f57777-4f6f-4c66-9fbe-bc91e2c34893 None None] MFA Rules not processed for user `b8ecd0b57d4f4d238eae8a3adb67dfe5`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:48:47.410047 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-01f57777-4f6f-4c66-9fbe-bc91e2c34893 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:48:47.499640 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1434/2843] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:48:47 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:48:47.504593 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-516ac638-e139-42dd-b74c-37bfdc025293 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:47.582564 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-516ac638-e139-42dd-b74c-37bfdc025293 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:47.588341 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fd077f2-0602-4d5e-a60c-1e557d0a9ecc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:47.589426 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fd077f2-0602-4d5e-a60c-1e557d0a9ecc admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:47.589807 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fd077f2-0602-4d5e-a60c-1e557d0a9ecc admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:47.590315 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fd077f2-0602-4d5e-a60c-1e557d0a9ecc admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:47.591283 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fd077f2-0602-4d5e-a60c-1e557d0a9ecc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:47.594575 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fd077f2-0602-4d5e-a60c-1e557d0a9ecc admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:47.633517 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1410/2844] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:48:47 2022] POST /identity/v3/projects => generated 383 bytes in 127 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:48:47.635215 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01f57777-4f6f-4c66-9fbe-bc91e2c34893 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:47.650577 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01f57777-4f6f-4c66-9fbe-bc91e2c34893 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:47.727520 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ffe5a33-4056-4770-80e1-624865814eac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:47.728344 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ffe5a33-4056-4770-80e1-624865814eac admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:47.728654 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ffe5a33-4056-4770-80e1-624865814eac admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:47.729000 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ffe5a33-4056-4770-80e1-624865814eac admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:47.729573 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ffe5a33-4056-4770-80e1-624865814eac admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:47.731473 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ffe5a33-4056-4770-80e1-624865814eac admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:47.758499 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1435/2845] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:48:47 2022] POST /identity/v3/users => generated 292 bytes in 124 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:48:47.763010 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fd077f2-0602-4d5e-a60c-1e557d0a9ecc admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:47.776584 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fd077f2-0602-4d5e-a60c-1e557d0a9ecc admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:47.782278 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-141f173f-8ad8-4cf2-b1c8-32cacdb45a3c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:47.782893 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-141f173f-8ad8-4cf2-b1c8-32cacdb45a3c admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:47.783174 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-141f173f-8ad8-4cf2-b1c8-32cacdb45a3c admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:47.783444 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-141f173f-8ad8-4cf2-b1c8-32cacdb45a3c admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:47.784745 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-141f173f-8ad8-4cf2-b1c8-32cacdb45a3c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:47.786842 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-141f173f-8ad8-4cf2-b1c8-32cacdb45a3c admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:47.800720 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1411/2846] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:48:47 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:48:47.804540 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ffe5a33-4056-4770-80e1-624865814eac admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:47.818447 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ffe5a33-4056-4770-80e1-624865814eac admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:47.824390 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40119ac0-15c9-457b-9369-2f22d14fcf0f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:47.825115 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40119ac0-15c9-457b-9369-2f22d14fcf0f admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:47.825459 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40119ac0-15c9-457b-9369-2f22d14fcf0f admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:47.825800 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40119ac0-15c9-457b-9369-2f22d14fcf0f admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:47.826664 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40119ac0-15c9-457b-9369-2f22d14fcf0f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:47.830099 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40119ac0-15c9-457b-9369-2f22d14fcf0f admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:47.845631 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1436/2847] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:48:47 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:48:47.846817 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-141f173f-8ad8-4cf2-b1c8-32cacdb45a3c admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:47.853157 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:48:47.860475 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-141f173f-8ad8-4cf2-b1c8-32cacdb45a3c admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:47.872275 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-138956ce-12d9-4920-991c-ae9339c0622a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:47.872994 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-138956ce-12d9-4920-991c-ae9339c0622a admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:47.873277 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-138956ce-12d9-4920-991c-ae9339c0622a admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:47.873554 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-138956ce-12d9-4920-991c-ae9339c0622a admin admin] PATH_INFO: `/v3/projects/fdb002a3d32c454fbb0922df72fb95f4/users/9cce91eac381490981a8c5c1aa1d99c0/roles/8f6cc574056c4278bc9778f14dc4b411` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:47.893027 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-138956ce-12d9-4920-991c-ae9339c0622a admin admin] RBAC: Authorizing `identity:create_grant(project_id=fdb002a3d32c454fbb0922df72fb95f4, user_id=9cce91eac381490981a8c5c1aa1d99c0, role_id=8f6cc574056c4278bc9778f14dc4b411)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:47.896188 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-138956ce-12d9-4920-991c-ae9339c0622a admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:47.912268 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1412/2848] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:48:47 2022] PUT /identity/v3/projects/fdb002a3d32c454fbb0922df72fb95f4/users/9cce91eac381490981a8c5c1aa1d99c0/roles/8f6cc574056c4278bc9778f14dc4b411 => generated 0 bytes in 66 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:48:47.924604 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75dc4de0-718f-426f-a891-7fdf2462fb03 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:47.924604 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75dc4de0-718f-426f-a891-7fdf2462fb03 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:47.925321 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75dc4de0-718f-426f-a891-7fdf2462fb03 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:47.951045 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:48:47.962559 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-75dc4de0-718f-426f-a891-7fdf2462fb03 None None] MFA Rules not processed for user `9cce91eac381490981a8c5c1aa1d99c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:48:47.965002 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-75dc4de0-718f-426f-a891-7fdf2462fb03 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:48:48.049272 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1437/2849] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:48:47 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:48:48.050445 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-138956ce-12d9-4920-991c-ae9339c0622a admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:48.129817 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-138956ce-12d9-4920-991c-ae9339c0622a admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:48.134843 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d0554d7-cf69-4b63-a6be-ce3c8ffbae27 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:48.135471 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d0554d7-cf69-4b63-a6be-ce3c8ffbae27 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:48.135750 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d0554d7-cf69-4b63-a6be-ce3c8ffbae27 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:48.136020 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d0554d7-cf69-4b63-a6be-ce3c8ffbae27 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:48.136685 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d0554d7-cf69-4b63-a6be-ce3c8ffbae27 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:48.139069 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d0554d7-cf69-4b63-a6be-ce3c8ffbae27 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:48.178294 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1413/2850] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:48:48 2022] POST /identity/v3/projects => generated 385 bytes in 127 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:48:48.179433 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75dc4de0-718f-426f-a891-7fdf2462fb03 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:48.191673 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75dc4de0-718f-426f-a891-7fdf2462fb03 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:48.268381 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09969368-20f2-44be-b148-6d762519e182 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:48.269134 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09969368-20f2-44be-b148-6d762519e182 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:48.269488 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09969368-20f2-44be-b148-6d762519e182 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:48.269833 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09969368-20f2-44be-b148-6d762519e182 admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:48.270546 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09969368-20f2-44be-b148-6d762519e182 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:48.272548 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09969368-20f2-44be-b148-6d762519e182 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:48.301830 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1438/2851] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:48:48 2022] POST /identity/v3/users => generated 299 bytes in 120 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:48:48.303452 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d0554d7-cf69-4b63-a6be-ce3c8ffbae27 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:48.315129 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d0554d7-cf69-4b63-a6be-ce3c8ffbae27 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:48.320375 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b2c899f-acc5-4274-8652-06789eb2a279 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:48.321241 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b2c899f-acc5-4274-8652-06789eb2a279 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:48.321634 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b2c899f-acc5-4274-8652-06789eb2a279 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:48.321991 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b2c899f-acc5-4274-8652-06789eb2a279 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:48.322708 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b2c899f-acc5-4274-8652-06789eb2a279 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:48.324707 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b2c899f-acc5-4274-8652-06789eb2a279 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:48.338564 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1414/2852] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:48:48 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:48:48.343208 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09969368-20f2-44be-b148-6d762519e182 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:48.356019 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09969368-20f2-44be-b148-6d762519e182 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:48.361848 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10939ea7-2d1a-47f5-8484-10ce3e63a556 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:48.362629 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10939ea7-2d1a-47f5-8484-10ce3e63a556 admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:48.362978 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10939ea7-2d1a-47f5-8484-10ce3e63a556 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:48.363321 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10939ea7-2d1a-47f5-8484-10ce3e63a556 admin admin] PATH_INFO: `/v3/projects/cb0f08b8c1804a069dd5dd5b057a4c26/users/6f4d6f5797a24d708b106df560afd931/roles/c51fe7c5ad804040879acfb54269455f` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:48.383305 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10939ea7-2d1a-47f5-8484-10ce3e63a556 admin admin] RBAC: Authorizing `identity:create_grant(project_id=cb0f08b8c1804a069dd5dd5b057a4c26, user_id=6f4d6f5797a24d708b106df560afd931, role_id=c51fe7c5ad804040879acfb54269455f)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:48.386107 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10939ea7-2d1a-47f5-8484-10ce3e63a556 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:48.404343 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1439/2853] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:48:48 2022] PUT /identity/v3/projects/cb0f08b8c1804a069dd5dd5b057a4c26/users/6f4d6f5797a24d708b106df560afd931/roles/c51fe7c5ad804040879acfb54269455f => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:48:48.408472 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b2c899f-acc5-4274-8652-06789eb2a279 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:48.419371 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b2c899f-acc5-4274-8652-06789eb2a279 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:48.514497 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbb41588-5aba-4325-a489-9585c1b7ffa6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:48.515429 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb41588-5aba-4325-a489-9585c1b7ffa6 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:48.515795 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb41588-5aba-4325-a489-9585c1b7ffa6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:48.516129 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb41588-5aba-4325-a489-9585c1b7ffa6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:48.516808 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbb41588-5aba-4325-a489-9585c1b7ffa6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:48.518502 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbb41588-5aba-4325-a489-9585c1b7ffa6 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:48.532270 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1415/2854] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:48:48 2022] GET /identity/v3/roles => generated 1420 bytes in 124 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 25 04:48:48.536700 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10939ea7-2d1a-47f5-8484-10ce3e63a556 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:48.548629 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10939ea7-2d1a-47f5-8484-10ce3e63a556 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:48.556163 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c860110-4898-48a3-9763-4a410aca6490 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:48.556957 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c860110-4898-48a3-9763-4a410aca6490 admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:48.557253 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c860110-4898-48a3-9763-4a410aca6490 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:48.557525 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c860110-4898-48a3-9763-4a410aca6490 admin admin] PATH_INFO: `/v3/projects/cb0f08b8c1804a069dd5dd5b057a4c26/users/6f4d6f5797a24d708b106df560afd931/roles/c51fe7c5ad804040879acfb54269455f` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:48.559468 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c860110-4898-48a3-9763-4a410aca6490 admin admin] RBAC: Authorizing `identity:create_grant(project_id=cb0f08b8c1804a069dd5dd5b057a4c26, user_id=6f4d6f5797a24d708b106df560afd931, role_id=c51fe7c5ad804040879acfb54269455f)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:48.562705 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c860110-4898-48a3-9763-4a410aca6490 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:48.576423 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1440/2855] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:48:48 2022] PUT /identity/v3/projects/cb0f08b8c1804a069dd5dd5b057a4c26/users/6f4d6f5797a24d708b106df560afd931/roles/c51fe7c5ad804040879acfb54269455f => generated 0 bytes in 40 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:48:48.587302 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04081663-e94c-4f20-8327-b2c7fc311ef7 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:48.587616 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04081663-e94c-4f20-8327-b2c7fc311ef7 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:48.587902 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04081663-e94c-4f20-8327-b2c7fc311ef7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:48.626904 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-04081663-e94c-4f20-8327-b2c7fc311ef7 None None] MFA Rules not processed for user `6f4d6f5797a24d708b106df560afd931`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:48:48.628849 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-04081663-e94c-4f20-8327-b2c7fc311ef7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:48:48.687892 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:48:48.735620 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1416/2856] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:48:48 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:48:48.745154 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c860110-4898-48a3-9763-4a410aca6490 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:48.748452 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-7c860110-4898-48a3-9763-4a410aca6490 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:48:48.835793 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c860110-4898-48a3-9763-4a410aca6490 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:48.841542 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7345bc3e-b1fa-47bf-8754-b80abf7a9ef9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ffbc136c77b84e8c87128f7247f4ec03', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:48.842177 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7345bc3e-b1fa-47bf-8754-b80abf7a9ef9 service cinder] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:48.842453 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7345bc3e-b1fa-47bf-8754-b80abf7a9ef9 service cinder] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:48.842721 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7345bc3e-b1fa-47bf-8754-b80abf7a9ef9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:48.850206 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:48:48.923184 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7345bc3e-b1fa-47bf-8754-b80abf7a9ef9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:48.924873 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7345bc3e-b1fa-47bf-8754-b80abf7a9ef9 service cinder] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:48.936023 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1441/2857] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:48:48 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 192 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:48:50.753225 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04081663-e94c-4f20-8327-b2c7fc311ef7 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:50.755430 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-04081663-e94c-4f20-8327-b2c7fc311ef7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:48:50.778595 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04081663-e94c-4f20-8327-b2c7fc311ef7 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:50.788350 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fe2fe73-afb2-4a83-9b76-2e68235a7e46 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ffbc136c77b84e8c87128f7247f4ec03', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:50.789117 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fe2fe73-afb2-4a83-9b76-2e68235a7e46 service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:50.789398 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fe2fe73-afb2-4a83-9b76-2e68235a7e46 service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:50.789672 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fe2fe73-afb2-4a83-9b76-2e68235a7e46 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:50.947547 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fe2fe73-afb2-4a83-9b76-2e68235a7e46 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:50.949064 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fe2fe73-afb2-4a83-9b76-2e68235a7e46 service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:50.971508 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1417/2858] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:48:50 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 219 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:48:52.079354 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7345bc3e-b1fa-47bf-8754-b80abf7a9ef9 service cinder] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:52.090222 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7345bc3e-b1fa-47bf-8754-b80abf7a9ef9 service cinder] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:52.097891 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e945fb8-0df6-4b79-b857-2f7381b9998f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ffbc136c77b84e8c87128f7247f4ec03', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:52.099220 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e945fb8-0df6-4b79-b857-2f7381b9998f service cinder] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:52.100144 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e945fb8-0df6-4b79-b857-2f7381b9998f service cinder] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:52.110876 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e945fb8-0df6-4b79-b857-2f7381b9998f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:52.124691 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e945fb8-0df6-4b79-b857-2f7381b9998f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:52.126211 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e945fb8-0df6-4b79-b857-2f7381b9998f service cinder] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:52.129781 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e13ec68-5400-4fb8-8dd5-611b1c7908c9 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:52.130602 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e13ec68-5400-4fb8-8dd5-611b1c7908c9 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:52.131224 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e13ec68-5400-4fb8-8dd5-611b1c7908c9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:52.136670 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1442/2859] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:48:52 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 58 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:48:52.174772 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-3e13ec68-5400-4fb8-8dd5-611b1c7908c9 None None] MFA Rules not processed for user `ef85cd347e6546229f92a41b74e3eb5d`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:48:52.177139 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-3e13ec68-5400-4fb8-8dd5-611b1c7908c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:48:52.296923 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1418/2860] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:48:52 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 04:48:52.313462 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e945fb8-0df6-4b79-b857-2f7381b9998f service cinder] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:52.315357 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-1e945fb8-0df6-4b79-b857-2f7381b9998f service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:48:52.404269 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e945fb8-0df6-4b79-b857-2f7381b9998f service cinder] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:52.411228 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c672511-2fb0-47dc-977d-65dc2231264a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e703b42c53974daaa608b0ccc3f018b1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'ResellerAdmin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:52.411892 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c672511-2fb0-47dc-977d-65dc2231264a service nova] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:52.412174 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c672511-2fb0-47dc-977d-65dc2231264a service nova] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:52.412468 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c672511-2fb0-47dc-977d-65dc2231264a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:52.414294 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-3c672511-2fb0-47dc-977d-65dc2231264a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:48:52.490959 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c672511-2fb0-47dc-977d-65dc2231264a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:52.492543 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c672511-2fb0-47dc-977d-65dc2231264a service nova] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:52.505967 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1443/2861] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:48:52 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 193 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:48:53.577851 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abcbd35f-6443-4752-aba9-e25d4e6ffecc None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:53.578874 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abcbd35f-6443-4752-aba9-e25d4e6ffecc None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:53.579498 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abcbd35f-6443-4752-aba9-e25d4e6ffecc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:53.603199 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-abcbd35f-6443-4752-aba9-e25d4e6ffecc None None] MFA Rules not processed for user `673b4f55a335484d927d7966d72690ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:48:53.604509 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-abcbd35f-6443-4752-aba9-e25d4e6ffecc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:48:53.618369 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1419/2862] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:48:53 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:48:56.197382 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c672511-2fb0-47dc-977d-65dc2231264a service nova] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:56.229199 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=100 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:48:56.262663 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c672511-2fb0-47dc-977d-65dc2231264a service nova] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:56.267521 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce7a3c60-5f13-4367-afca-c7e466b087df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:56.268145 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce7a3c60-5f13-4367-afca-c7e466b087df admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:56.268421 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce7a3c60-5f13-4367-afca-c7e466b087df admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:56.268725 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce7a3c60-5f13-4367-afca-c7e466b087df admin admin] PATH_INFO: `/v3/users/b8ecd0b57d4f4d238eae8a3adb67dfe5` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:56.269841 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce7a3c60-5f13-4367-afca-c7e466b087df admin admin] RBAC: Authorizing `identity:delete_user(user_id=b8ecd0b57d4f4d238eae8a3adb67dfe5)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:56.271760 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce7a3c60-5f13-4367-afca-c7e466b087df admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:56.320451 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-ce7a3c60-5f13-4367-afca-c7e466b087df admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b8ecd0b57d4f4d238eae8a3adb67dfe5'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:56.327914 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-ce7a3c60-5f13-4367-afca-c7e466b087df admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b8ecd0b57d4f4d238eae8a3adb67dfe5'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:56.340696 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-ce7a3c60-5f13-4367-afca-c7e466b087df admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b8ecd0b57d4f4d238eae8a3adb67dfe5'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:56.341418 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-ce7a3c60-5f13-4367-afca-c7e466b087df admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b8ecd0b57d4f4d238eae8a3adb67dfe5'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:56.348732 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-ce7a3c60-5f13-4367-afca-c7e466b087df admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b8ecd0b57d4f4d238eae8a3adb67dfe5'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:56.354603 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1444/2863] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:48:56 2022] DELETE /identity/v3/users/b8ecd0b57d4f4d238eae8a3adb67dfe5 => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:48:56.359732 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aeb53e2-6a6f-42d8-8fa9-ff9a6b5be831 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:56.360254 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aeb53e2-6a6f-42d8-8fa9-ff9a6b5be831 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:56.360422 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aeb53e2-6a6f-42d8-8fa9-ff9a6b5be831 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:56.379847 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-1aeb53e2-6a6f-42d8-8fa9-ff9a6b5be831 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:48:56.381110 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-1aeb53e2-6a6f-42d8-8fa9-ff9a6b5be831 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:48:56.426285 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:48:56.441821 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1420/2864] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:48:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:48:56.451724 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce7a3c60-5f13-4367-afca-c7e466b087df admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:56.453324 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-ce7a3c60-5f13-4367-afca-c7e466b087df admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:48:56.514513 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce7a3c60-5f13-4367-afca-c7e466b087df admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:56.518853 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54eac0cc-ff16-4439-939a-8ef5cc8cf479 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:56.519586 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54eac0cc-ff16-4439-939a-8ef5cc8cf479 service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:56.519947 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54eac0cc-ff16-4439-939a-8ef5cc8cf479 service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:56.520267 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54eac0cc-ff16-4439-939a-8ef5cc8cf479 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:56.529679 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54eac0cc-ff16-4439-939a-8ef5cc8cf479 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:56.531633 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54eac0cc-ff16-4439-939a-8ef5cc8cf479 service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:56.541624 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1445/2865] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:48:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:48:56.818333 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aeb53e2-6a6f-42d8-8fa9-ff9a6b5be831 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:56.819815 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-1aeb53e2-6a6f-42d8-8fa9-ff9a6b5be831 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:48:56.834079 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aeb53e2-6a6f-42d8-8fa9-ff9a6b5be831 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:56.838729 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44a22664-678d-43f1-b0b8-246b68eadc59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:56.840067 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44a22664-678d-43f1-b0b8-246b68eadc59 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:56.840347 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44a22664-678d-43f1-b0b8-246b68eadc59 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:56.840610 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44a22664-678d-43f1-b0b8-246b68eadc59 admin admin] PATH_INFO: `/v3/projects/926a6ac0c3664ef6840c4ab8873896e0` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:56.841907 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44a22664-678d-43f1-b0b8-246b68eadc59 admin admin] RBAC: Authorizing `identity:delete_project(project_id=926a6ac0c3664ef6840c4ab8873896e0)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:56.844891 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44a22664-678d-43f1-b0b8-246b68eadc59 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:56.896806 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-44a22664-678d-43f1-b0b8-246b68eadc59 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '926a6ac0c3664ef6840c4ab8873896e0', 'request_id': 'req-44a22664-678d-43f1-b0b8-246b68eadc59'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:56.904600 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-44a22664-678d-43f1-b0b8-246b68eadc59 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '926a6ac0c3664ef6840c4ab8873896e0', 'request_id': 'req-44a22664-678d-43f1-b0b8-246b68eadc59'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:56.910938 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-44a22664-678d-43f1-b0b8-246b68eadc59 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '926a6ac0c3664ef6840c4ab8873896e0', 'request_id': 'req-44a22664-678d-43f1-b0b8-246b68eadc59'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:56.923120 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-44a22664-678d-43f1-b0b8-246b68eadc59 admin admin] The token cache is being invalidate because project 926a6ac0c3664ef6840c4ab8873896e0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:48:56.923413 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-44a22664-678d-43f1-b0b8-246b68eadc59 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:56.924081 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-44a22664-678d-43f1-b0b8-246b68eadc59 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:56.925602 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1421/2866] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:48:56 2022] DELETE /identity/v3/projects/926a6ac0c3664ef6840c4ab8873896e0 => generated 0 bytes in 108 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:48:56.929987 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54eac0cc-ff16-4439-939a-8ef5cc8cf479 service neutron] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:56.939601 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54eac0cc-ff16-4439-939a-8ef5cc8cf479 service neutron] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:56.992516 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:48:57.013895 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68f6b90b-43c3-404e-8998-f1ba6811627b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:57.014555 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68f6b90b-43c3-404e-8998-f1ba6811627b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:57.014832 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68f6b90b-43c3-404e-8998-f1ba6811627b admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:57.015102 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68f6b90b-43c3-404e-8998-f1ba6811627b admin admin] PATH_INFO: `/v3/users/9cce91eac381490981a8c5c1aa1d99c0` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:57.021517 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68f6b90b-43c3-404e-8998-f1ba6811627b admin admin] RBAC: Authorizing `identity:delete_user(user_id=9cce91eac381490981a8c5c1aa1d99c0)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:57.023710 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68f6b90b-43c3-404e-8998-f1ba6811627b admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:57.072895 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-68f6b90b-43c3-404e-8998-f1ba6811627b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9cce91eac381490981a8c5c1aa1d99c0'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:57.081159 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-68f6b90b-43c3-404e-8998-f1ba6811627b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9cce91eac381490981a8c5c1aa1d99c0'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:57.096256 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-68f6b90b-43c3-404e-8998-f1ba6811627b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9cce91eac381490981a8c5c1aa1d99c0'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:57.097114 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-68f6b90b-43c3-404e-8998-f1ba6811627b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9cce91eac381490981a8c5c1aa1d99c0'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:57.104727 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-68f6b90b-43c3-404e-8998-f1ba6811627b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9cce91eac381490981a8c5c1aa1d99c0'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:57.111550 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1446/2867] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:48:56 2022] DELETE /identity/v3/users/9cce91eac381490981a8c5c1aa1d99c0 => generated 0 bytes in 182 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:48:57.401078 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44a22664-678d-43f1-b0b8-246b68eadc59 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:57.402802 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-44a22664-678d-43f1-b0b8-246b68eadc59 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:48:57.417419 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44a22664-678d-43f1-b0b8-246b68eadc59 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:57.482478 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e019dc0d-1815-4b1a-88e5-5225edcbf593 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:57.483106 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e019dc0d-1815-4b1a-88e5-5225edcbf593 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:57.483387 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e019dc0d-1815-4b1a-88e5-5225edcbf593 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:57.483659 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e019dc0d-1815-4b1a-88e5-5225edcbf593 admin admin] PATH_INFO: `/v3/projects/fdb002a3d32c454fbb0922df72fb95f4` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:57.484935 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e019dc0d-1815-4b1a-88e5-5225edcbf593 admin admin] RBAC: Authorizing `identity:delete_project(project_id=fdb002a3d32c454fbb0922df72fb95f4)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:57.486800 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e019dc0d-1815-4b1a-88e5-5225edcbf593 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:57.535227 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-e019dc0d-1815-4b1a-88e5-5225edcbf593 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fdb002a3d32c454fbb0922df72fb95f4', 'request_id': 'req-e019dc0d-1815-4b1a-88e5-5225edcbf593'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:57.542749 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-e019dc0d-1815-4b1a-88e5-5225edcbf593 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fdb002a3d32c454fbb0922df72fb95f4', 'request_id': 'req-e019dc0d-1815-4b1a-88e5-5225edcbf593'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:57.548506 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-e019dc0d-1815-4b1a-88e5-5225edcbf593 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fdb002a3d32c454fbb0922df72fb95f4', 'request_id': 'req-e019dc0d-1815-4b1a-88e5-5225edcbf593'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:57.559965 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-e019dc0d-1815-4b1a-88e5-5225edcbf593 admin admin] The token cache is being invalidate because project fdb002a3d32c454fbb0922df72fb95f4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:48:57.560256 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-e019dc0d-1815-4b1a-88e5-5225edcbf593 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:57.560937 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-e019dc0d-1815-4b1a-88e5-5225edcbf593 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:57.562945 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1422/2868] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:48:57 2022] DELETE /identity/v3/projects/fdb002a3d32c454fbb0922df72fb95f4 => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:48:57.567110 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68f6b90b-43c3-404e-8998-f1ba6811627b admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:57.568522 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-68f6b90b-43c3-404e-8998-f1ba6811627b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:48:57.587221 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68f6b90b-43c3-404e-8998-f1ba6811627b admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:57.654451 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a7e3d80-6e7f-45de-9107-69ece76f9781 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:57.655315 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a7e3d80-6e7f-45de-9107-69ece76f9781 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:57.655874 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a7e3d80-6e7f-45de-9107-69ece76f9781 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:57.656354 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a7e3d80-6e7f-45de-9107-69ece76f9781 admin admin] PATH_INFO: `/v3/users/6f4d6f5797a24d708b106df560afd931` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:57.659294 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a7e3d80-6e7f-45de-9107-69ece76f9781 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6f4d6f5797a24d708b106df560afd931)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:57.660208 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a7e3d80-6e7f-45de-9107-69ece76f9781 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:57.713095 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-9a7e3d80-6e7f-45de-9107-69ece76f9781 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6f4d6f5797a24d708b106df560afd931'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:57.721291 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-9a7e3d80-6e7f-45de-9107-69ece76f9781 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6f4d6f5797a24d708b106df560afd931'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:57.733625 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-9a7e3d80-6e7f-45de-9107-69ece76f9781 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6f4d6f5797a24d708b106df560afd931'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:57.734760 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-9a7e3d80-6e7f-45de-9107-69ece76f9781 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6f4d6f5797a24d708b106df560afd931'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:57.735970 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=78,keystone:DELETE=20,keystone:INSERT=2 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:48:57.744926 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-9a7e3d80-6e7f-45de-9107-69ece76f9781 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6f4d6f5797a24d708b106df560afd931'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:57.752664 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1447/2869] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:48:57 2022] DELETE /identity/v3/users/6f4d6f5797a24d708b106df560afd931 => generated 0 bytes in 186 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:48:58.053584 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e019dc0d-1815-4b1a-88e5-5225edcbf593 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:58.053584 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-e019dc0d-1815-4b1a-88e5-5225edcbf593 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:48:58.063222 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:48:58.068893 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e019dc0d-1815-4b1a-88e5-5225edcbf593 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:58.124268 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a7e3d80-6e7f-45de-9107-69ece76f9781 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:58.129546 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-9a7e3d80-6e7f-45de-9107-69ece76f9781 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:48:58.236867 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a7e3d80-6e7f-45de-9107-69ece76f9781 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:58.242583 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5c330b3-404c-47d3-82d6-baa5e1a26a0a service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '676d04ef5a984edcaad63d9156568660', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:58.243227 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c330b3-404c-47d3-82d6-baa5e1a26a0a service placement] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:58.243533 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c330b3-404c-47d3-82d6-baa5e1a26a0a service placement] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:58.243784 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c330b3-404c-47d3-82d6-baa5e1a26a0a service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:58.246069 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-f5c330b3-404c-47d3-82d6-baa5e1a26a0a service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:48:58.263598 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5c330b3-404c-47d3-82d6-baa5e1a26a0a service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:58.265269 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5c330b3-404c-47d3-82d6-baa5e1a26a0a service placement] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:58.278328 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1448/2870] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:48:58 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 154 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:48:58.310003 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57f6b701-8423-4fb7-adbd-e4b5f77b242a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:58.310926 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57f6b701-8423-4fb7-adbd-e4b5f77b242a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:58.311279 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57f6b701-8423-4fb7-adbd-e4b5f77b242a admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:58.311629 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57f6b701-8423-4fb7-adbd-e4b5f77b242a admin admin] PATH_INFO: `/v3/projects/cb0f08b8c1804a069dd5dd5b057a4c26` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:58.313330 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57f6b701-8423-4fb7-adbd-e4b5f77b242a admin admin] RBAC: Authorizing `identity:delete_project(project_id=cb0f08b8c1804a069dd5dd5b057a4c26)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:58.316252 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57f6b701-8423-4fb7-adbd-e4b5f77b242a admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:58.379297 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-57f6b701-8423-4fb7-adbd-e4b5f77b242a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cb0f08b8c1804a069dd5dd5b057a4c26', 'request_id': 'req-57f6b701-8423-4fb7-adbd-e4b5f77b242a'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:58.390289 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-57f6b701-8423-4fb7-adbd-e4b5f77b242a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cb0f08b8c1804a069dd5dd5b057a4c26', 'request_id': 'req-57f6b701-8423-4fb7-adbd-e4b5f77b242a'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:58.397735 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-57f6b701-8423-4fb7-adbd-e4b5f77b242a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cb0f08b8c1804a069dd5dd5b057a4c26', 'request_id': 'req-57f6b701-8423-4fb7-adbd-e4b5f77b242a'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:58.411235 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-57f6b701-8423-4fb7-adbd-e4b5f77b242a admin admin] The token cache is being invalidate because project cb0f08b8c1804a069dd5dd5b057a4c26 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:48:58.412284 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-57f6b701-8423-4fb7-adbd-e4b5f77b242a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:58.412284 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-57f6b701-8423-4fb7-adbd-e4b5f77b242a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:48:58.415743 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1423/2871] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:48:58 2022] DELETE /identity/v3/projects/cb0f08b8c1804a069dd5dd5b057a4c26 => generated 0 bytes in 365 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:48:58.421404 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d1ccf66-7be2-44f9-8513-4ed554e2e2c1 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:58.422460 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d1ccf66-7be2-44f9-8513-4ed554e2e2c1 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:58.422771 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d1ccf66-7be2-44f9-8513-4ed554e2e2c1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:58.445480 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-1d1ccf66-7be2-44f9-8513-4ed554e2e2c1 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:48:58.446710 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-1d1ccf66-7be2-44f9-8513-4ed554e2e2c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:48:58.514032 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1449/2872] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:48:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:48:58.523008 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03b81307-e84a-48d7-a97a-b6d32a99a5ed None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:58.523400 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03b81307-e84a-48d7-a97a-b6d32a99a5ed None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:58.523777 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03b81307-e84a-48d7-a97a-b6d32a99a5ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:58.547823 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-03b81307-e84a-48d7-a97a-b6d32a99a5ed None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:48:58.549180 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-03b81307-e84a-48d7-a97a-b6d32a99a5ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:48:58.557985 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1424/2873] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:48:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:48:58.563142 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d1ccf66-7be2-44f9-8513-4ed554e2e2c1 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:58.573432 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d1ccf66-7be2-44f9-8513-4ed554e2e2c1 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:58.578245 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cc70715-d184-4716-ad6c-2440b74f538f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:58.578951 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc70715-d184-4716-ad6c-2440b74f538f admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:58.579235 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc70715-d184-4716-ad6c-2440b74f538f admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:58.579512 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc70715-d184-4716-ad6c-2440b74f538f admin admin] PATH_INFO: `/v3/domains` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:58.580328 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cc70715-d184-4716-ad6c-2440b74f538f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81434) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:48:58.580462 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cc70715-d184-4716-ad6c-2440b74f538f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:58.582101 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cc70715-d184-4716-ad6c-2440b74f538f admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:58.594610 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1450/2874] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Sep 25 04:48:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:48:58.598890 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03b81307-e84a-48d7-a97a-b6d32a99a5ed None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:58.608135 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03b81307-e84a-48d7-a97a-b6d32a99a5ed None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:58.613555 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a882ff70-f6db-4639-b22e-2f11f44adb59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:58.614171 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a882ff70-f6db-4639-b22e-2f11f44adb59 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:58.614448 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a882ff70-f6db-4639-b22e-2f11f44adb59 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:58.614719 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a882ff70-f6db-4639-b22e-2f11f44adb59 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:58.615359 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a882ff70-f6db-4639-b22e-2f11f44adb59 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:58.617298 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a882ff70-f6db-4639-b22e-2f11f44adb59 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:58.645273 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1425/2875] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:48:58 2022] POST /identity/v3/projects => generated 367 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:48:58.649642 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cc70715-d184-4716-ad6c-2440b74f538f admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:58.659812 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cc70715-d184-4716-ad6c-2440b74f538f admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:58.718421 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de64342f-fb75-4bd9-bd79-03a4a6699c79 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:58.718972 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de64342f-fb75-4bd9-bd79-03a4a6699c79 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:58.719255 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de64342f-fb75-4bd9-bd79-03a4a6699c79 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:58.719526 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de64342f-fb75-4bd9-bd79-03a4a6699c79 admin admin] PATH_INFO: `/v3/users` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:58.720170 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de64342f-fb75-4bd9-bd79-03a4a6699c79 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:58.722119 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de64342f-fb75-4bd9-bd79-03a4a6699c79 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:58.733199 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:48:58.747196 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1451/2876] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:48:58 2022] POST /identity/v3/users => generated 284 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 25 04:48:58.752285 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a882ff70-f6db-4639-b22e-2f11f44adb59 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:58.762543 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a882ff70-f6db-4639-b22e-2f11f44adb59 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:58.767998 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f7f362b-1996-4d95-9b8b-35e351603866 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:58.768656 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f7f362b-1996-4d95-9b8b-35e351603866 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:58.768951 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f7f362b-1996-4d95-9b8b-35e351603866 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:58.769219 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f7f362b-1996-4d95-9b8b-35e351603866 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:58.769858 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f7f362b-1996-4d95-9b8b-35e351603866 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:58.771456 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f7f362b-1996-4d95-9b8b-35e351603866 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:58.782554 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1426/2877] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:48:58 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:48:58.787763 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de64342f-fb75-4bd9-bd79-03a4a6699c79 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:58.800325 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de64342f-fb75-4bd9-bd79-03a4a6699c79 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:58.805193 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a6a4792-3392-4e3f-96e8-86882cb2a815 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:58.805812 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a6a4792-3392-4e3f-96e8-86882cb2a815 admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:58.806097 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a6a4792-3392-4e3f-96e8-86882cb2a815 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:58.806372 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a6a4792-3392-4e3f-96e8-86882cb2a815 admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:58.807018 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a6a4792-3392-4e3f-96e8-86882cb2a815 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:58.808618 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a6a4792-3392-4e3f-96e8-86882cb2a815 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:58.822132 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1452/2878] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:48:58 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:48:58.823936 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f7f362b-1996-4d95-9b8b-35e351603866 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:58.834082 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f7f362b-1996-4d95-9b8b-35e351603866 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:58.838692 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5889a39-a679-4415-bb04-f5760f12e203 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:58.839593 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5889a39-a679-4415-bb04-f5760f12e203 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:58.839875 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5889a39-a679-4415-bb04-f5760f12e203 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:58.840148 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5889a39-a679-4415-bb04-f5760f12e203 admin admin] PATH_INFO: `/v3/projects/3f92b29547464054b99e550f8eb56cc3/users/5bd35889f8ce4b739d64cb357f8e2f4c/roles/8f6cc574056c4278bc9778f14dc4b411` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:58.856543 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5889a39-a679-4415-bb04-f5760f12e203 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3f92b29547464054b99e550f8eb56cc3, user_id=5bd35889f8ce4b739d64cb357f8e2f4c, role_id=8f6cc574056c4278bc9778f14dc4b411)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:58.859844 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5889a39-a679-4415-bb04-f5760f12e203 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:58.872696 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1427/2879] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:48:58 2022] PUT /identity/v3/projects/3f92b29547464054b99e550f8eb56cc3/users/5bd35889f8ce4b739d64cb357f8e2f4c/roles/8f6cc574056c4278bc9778f14dc4b411 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:48:58.884130 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3769eff2-1534-4e8f-a91f-0fc9e8590870 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:58.884445 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3769eff2-1534-4e8f-a91f-0fc9e8590870 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:58.884762 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3769eff2-1534-4e8f-a91f-0fc9e8590870 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:58.914226 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-3769eff2-1534-4e8f-a91f-0fc9e8590870 None None] MFA Rules not processed for user `5bd35889f8ce4b739d64cb357f8e2f4c`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:48:58.915499 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-3769eff2-1534-4e8f-a91f-0fc9e8590870 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:48:58.985126 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1453/2880] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:48:58 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:48:58.997099 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5889a39-a679-4415-bb04-f5760f12e203 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:48:58.999160 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-e5889a39-a679-4415-bb04-f5760f12e203 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:48:59.034124 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:48:59.060732 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5889a39-a679-4415-bb04-f5760f12e203 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:48:59.064581 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea4f69f1-3608-4fe0-9d9e-6ca8ea7edba2 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '316eaa67d5464091af47efd489b96b87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:48:59.065224 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea4f69f1-3608-4fe0-9d9e-6ca8ea7edba2 service glance] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:48:59.065508 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea4f69f1-3608-4fe0-9d9e-6ca8ea7edba2 service glance] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:48:59.065783 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea4f69f1-3608-4fe0-9d9e-6ca8ea7edba2 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:48:59.074049 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea4f69f1-3608-4fe0-9d9e-6ca8ea7edba2 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:48:59.075473 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea4f69f1-3608-4fe0-9d9e-6ca8ea7edba2 service glance] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:48:59.085477 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1428/2881] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:48:58 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:49:08.981518 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=39 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:49:09.082275 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=12 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:49:18.712745 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3769eff2-1534-4e8f-a91f-0fc9e8590870 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:49:18.714432 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-3769eff2-1534-4e8f-a91f-0fc9e8590870 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:49:18.769607 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3769eff2-1534-4e8f-a91f-0fc9e8590870 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:49:18.773140 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-378aad39-d195-469e-bb40-41816a20a958 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:49:18.773753 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-378aad39-d195-469e-bb40-41816a20a958 service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:49:18.774016 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-378aad39-d195-469e-bb40-41816a20a958 service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:49:18.774284 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-378aad39-d195-469e-bb40-41816a20a958 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:49:18.775985 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-378aad39-d195-469e-bb40-41816a20a958 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:49:18.785984 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-378aad39-d195-469e-bb40-41816a20a958 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:49:18.787377 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-378aad39-d195-469e-bb40-41816a20a958 service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:49:18.795240 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1454/2882] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:49:18 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:49:20.264587 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea4f69f1-3608-4fe0-9d9e-6ca8ea7edba2 service glance] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:49:20.266076 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-ea4f69f1-3608-4fe0-9d9e-6ca8ea7edba2 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:49:20.337263 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea4f69f1-3608-4fe0-9d9e-6ca8ea7edba2 service glance] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:49:20.342538 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83aad5b8-a61f-4ed0-971b-c1f97b58d5b5 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '676d04ef5a984edcaad63d9156568660', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:49:20.343149 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83aad5b8-a61f-4ed0-971b-c1f97b58d5b5 service placement] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:49:20.343427 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83aad5b8-a61f-4ed0-971b-c1f97b58d5b5 service placement] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:49:20.343699 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83aad5b8-a61f-4ed0-971b-c1f97b58d5b5 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:49:20.345472 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-83aad5b8-a61f-4ed0-971b-c1f97b58d5b5 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:49:20.357712 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83aad5b8-a61f-4ed0-971b-c1f97b58d5b5 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:49:20.359319 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83aad5b8-a61f-4ed0-971b-c1f97b58d5b5 service placement] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:49:20.370522 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1429/2883] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:49:20 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:49:28.791622 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=26 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:49:30.366049 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=32 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:49:35.677701 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-378aad39-d195-469e-bb40-41816a20a958 service neutron] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:49:35.679297 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-378aad39-d195-469e-bb40-41816a20a958 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:49:35.759272 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-378aad39-d195-469e-bb40-41816a20a958 service neutron] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:49:35.765408 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89ecc704-f141-48ed-b207-f1c5bbadf1f6 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e703b42c53974daaa608b0ccc3f018b1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'ResellerAdmin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:49:35.766025 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89ecc704-f141-48ed-b207-f1c5bbadf1f6 service nova] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:49:35.766308 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89ecc704-f141-48ed-b207-f1c5bbadf1f6 service nova] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:49:35.766576 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89ecc704-f141-48ed-b207-f1c5bbadf1f6 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:49:35.768861 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-89ecc704-f141-48ed-b207-f1c5bbadf1f6 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:49:35.782882 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89ecc704-f141-48ed-b207-f1c5bbadf1f6 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:49:35.785261 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89ecc704-f141-48ed-b207-f1c5bbadf1f6 service nova] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:49:35.795826 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1455/2884] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:49:35 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 119 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:49:42.071977 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83aad5b8-a61f-4ed0-971b-c1f97b58d5b5 service placement] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:49:42.073399 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-83aad5b8-a61f-4ed0-971b-c1f97b58d5b5 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:49:42.085932 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83aad5b8-a61f-4ed0-971b-c1f97b58d5b5 service placement] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:49:42.090204 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3172cba1-bcc3-4adf-9958-f06f8403cfa5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:49:42.090799 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3172cba1-bcc3-4adf-9958-f06f8403cfa5 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:49:42.091078 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3172cba1-bcc3-4adf-9958-f06f8403cfa5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:49:42.091348 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3172cba1-bcc3-4adf-9958-f06f8403cfa5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:49:42.093138 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-3172cba1-bcc3-4adf-9958-f06f8403cfa5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:49:42.155847 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3172cba1-bcc3-4adf-9958-f06f8403cfa5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:49:42.157321 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3172cba1-bcc3-4adf-9958-f06f8403cfa5 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:49:42.166806 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1430/2885] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:49:42 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:49:43.971329 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89ecc704-f141-48ed-b207-f1c5bbadf1f6 service nova] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:49:43.983148 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89ecc704-f141-48ed-b207-f1c5bbadf1f6 service nova] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:49:43.989124 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad2f136d-6e47-407d-8232-7c4627facbd9 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e703b42c53974daaa608b0ccc3f018b1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'ResellerAdmin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:49:43.989865 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad2f136d-6e47-407d-8232-7c4627facbd9 service nova] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:49:43.990157 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad2f136d-6e47-407d-8232-7c4627facbd9 service nova] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:49:43.990426 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad2f136d-6e47-407d-8232-7c4627facbd9 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:49:43.992191 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-ad2f136d-6e47-407d-8232-7c4627facbd9 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:49:44.005908 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad2f136d-6e47-407d-8232-7c4627facbd9 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:49:44.007509 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad2f136d-6e47-407d-8232-7c4627facbd9 service nova] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:49:44.019873 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1456/2886] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:49:43 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 50 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:49:46.281274 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3172cba1-bcc3-4adf-9958-f06f8403cfa5 service neutron] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:49:46.347155 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3172cba1-bcc3-4adf-9958-f06f8403cfa5 service neutron] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:49:46.351715 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-978e437d-ff0c-4804-b002-eb73c1d43a92 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:49:46.352331 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-978e437d-ff0c-4804-b002-eb73c1d43a92 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:49:46.352608 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-978e437d-ff0c-4804-b002-eb73c1d43a92 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:49:46.352905 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-978e437d-ff0c-4804-b002-eb73c1d43a92 admin admin] PATH_INFO: `/v3/users/5bd35889f8ce4b739d64cb357f8e2f4c` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:49:46.354121 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-978e437d-ff0c-4804-b002-eb73c1d43a92 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5bd35889f8ce4b739d64cb357f8e2f4c)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:49:46.356209 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-978e437d-ff0c-4804-b002-eb73c1d43a92 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:49:46.402260 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-978e437d-ff0c-4804-b002-eb73c1d43a92 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5bd35889f8ce4b739d64cb357f8e2f4c'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:49:46.409574 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-978e437d-ff0c-4804-b002-eb73c1d43a92 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5bd35889f8ce4b739d64cb357f8e2f4c'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:49:46.421561 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-978e437d-ff0c-4804-b002-eb73c1d43a92 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5bd35889f8ce4b739d64cb357f8e2f4c'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:49:46.422195 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-978e437d-ff0c-4804-b002-eb73c1d43a92 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5bd35889f8ce4b739d64cb357f8e2f4c'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:49:46.429751 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-978e437d-ff0c-4804-b002-eb73c1d43a92 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5bd35889f8ce4b739d64cb357f8e2f4c'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:49:46.435482 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1431/2887] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:49:46 2022] DELETE /identity/v3/users/5bd35889f8ce4b739d64cb357f8e2f4c => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:49:46.440306 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7be289cc-c7ec-470a-8a04-44d6d6976b04 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:49:46.440805 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7be289cc-c7ec-470a-8a04-44d6d6976b04 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:49:46.440906 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7be289cc-c7ec-470a-8a04-44d6d6976b04 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:49:46.459595 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-7be289cc-c7ec-470a-8a04-44d6d6976b04 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:49:46.460857 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-7be289cc-c7ec-470a-8a04-44d6d6976b04 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:49:46.519441 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1457/2888] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:49:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:49:46.529032 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-978e437d-ff0c-4804-b002-eb73c1d43a92 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:49:46.530419 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-978e437d-ff0c-4804-b002-eb73c1d43a92 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:49:46.545157 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:49:46.589766 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-978e437d-ff0c-4804-b002-eb73c1d43a92 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:49:46.593418 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2da9b520-0b3b-4682-977e-1e8337cadf6a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:49:46.594019 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2da9b520-0b3b-4682-977e-1e8337cadf6a service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:49:46.594297 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2da9b520-0b3b-4682-977e-1e8337cadf6a service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:49:46.594568 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2da9b520-0b3b-4682-977e-1e8337cadf6a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:49:46.604076 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2da9b520-0b3b-4682-977e-1e8337cadf6a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:49:46.605515 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2da9b520-0b3b-4682-977e-1e8337cadf6a service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:49:46.615352 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1432/2889] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:49:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:49:46.883488 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7be289cc-c7ec-470a-8a04-44d6d6976b04 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:49:46.885109 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-7be289cc-c7ec-470a-8a04-44d6d6976b04 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:49:46.900153 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7be289cc-c7ec-470a-8a04-44d6d6976b04 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:49:46.904939 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22e20db9-5b64-49de-8209-61eb86d4dc7d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:49:46.905578 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22e20db9-5b64-49de-8209-61eb86d4dc7d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:49:46.906385 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22e20db9-5b64-49de-8209-61eb86d4dc7d admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:49:46.906665 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22e20db9-5b64-49de-8209-61eb86d4dc7d admin admin] PATH_INFO: `/v3/projects/3f92b29547464054b99e550f8eb56cc3` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:49:46.908621 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22e20db9-5b64-49de-8209-61eb86d4dc7d admin admin] RBAC: Authorizing `identity:delete_project(project_id=3f92b29547464054b99e550f8eb56cc3)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:49:46.910835 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22e20db9-5b64-49de-8209-61eb86d4dc7d admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:49:46.957360 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-22e20db9-5b64-49de-8209-61eb86d4dc7d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3f92b29547464054b99e550f8eb56cc3', 'request_id': 'req-22e20db9-5b64-49de-8209-61eb86d4dc7d'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:49:46.964796 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-22e20db9-5b64-49de-8209-61eb86d4dc7d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3f92b29547464054b99e550f8eb56cc3', 'request_id': 'req-22e20db9-5b64-49de-8209-61eb86d4dc7d'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:49:46.970289 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-22e20db9-5b64-49de-8209-61eb86d4dc7d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3f92b29547464054b99e550f8eb56cc3', 'request_id': 'req-22e20db9-5b64-49de-8209-61eb86d4dc7d'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:49:46.973037 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:49:46.983132 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-22e20db9-5b64-49de-8209-61eb86d4dc7d admin admin] The token cache is being invalidate because project 3f92b29547464054b99e550f8eb56cc3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81434) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:49:46.983588 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-22e20db9-5b64-49de-8209-61eb86d4dc7d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:49:46.984550 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-22e20db9-5b64-49de-8209-61eb86d4dc7d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:49:46.986089 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1458/2890] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:49:46 2022] DELETE /identity/v3/projects/3f92b29547464054b99e550f8eb56cc3 => generated 0 bytes in 103 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:49:46.991124 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee8bb799-8358-4466-acd9-474d79720ed7 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:49:46.991409 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee8bb799-8358-4466-acd9-474d79720ed7 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:49:46.991837 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee8bb799-8358-4466-acd9-474d79720ed7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:49:47.011119 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-ee8bb799-8358-4466-acd9-474d79720ed7 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:49:47.012330 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-ee8bb799-8358-4466-acd9-474d79720ed7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:49:47.070105 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1433/2891] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:49:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:49:47.077940 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-792121fd-a638-4287-abb7-c92432bbfa6a None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:49:47.078227 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-792121fd-a638-4287-abb7-c92432bbfa6a None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:49:47.078502 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-792121fd-a638-4287-abb7-c92432bbfa6a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:49:47.099602 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-792121fd-a638-4287-abb7-c92432bbfa6a None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:49:47.100829 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-792121fd-a638-4287-abb7-c92432bbfa6a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:49:47.108535 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1459/2892] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:49:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:49:47.112916 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee8bb799-8358-4466-acd9-474d79720ed7 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:49:47.122306 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee8bb799-8358-4466-acd9-474d79720ed7 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:49:47.127454 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64fa570c-ecb7-4389-8e76-c6ea3dac319f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:49:47.128272 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64fa570c-ecb7-4389-8e76-c6ea3dac319f admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:49:47.128552 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64fa570c-ecb7-4389-8e76-c6ea3dac319f admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:49:47.128851 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64fa570c-ecb7-4389-8e76-c6ea3dac319f admin admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:49:47.129500 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64fa570c-ecb7-4389-8e76-c6ea3dac319f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:49:47.129785 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64fa570c-ecb7-4389-8e76-c6ea3dac319f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:49:47.131350 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64fa570c-ecb7-4389-8e76-c6ea3dac319f admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:49:47.143243 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1434/2893] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Sep 25 04:49:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:49:47.147413 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-792121fd-a638-4287-abb7-c92432bbfa6a None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:49:47.156578 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-792121fd-a638-4287-abb7-c92432bbfa6a None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:49:47.161361 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fc6baf4-36a9-495d-85ec-2a7ef3b05a3e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:49:47.161952 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fc6baf4-36a9-495d-85ec-2a7ef3b05a3e admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:49:47.162230 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fc6baf4-36a9-495d-85ec-2a7ef3b05a3e admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:49:47.162502 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fc6baf4-36a9-495d-85ec-2a7ef3b05a3e admin admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:49:47.163134 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fc6baf4-36a9-495d-85ec-2a7ef3b05a3e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:49:47.165055 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fc6baf4-36a9-495d-85ec-2a7ef3b05a3e admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:49:47.192458 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1460/2894] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:49:47 2022] POST /identity/v3/projects => generated 383 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:49:47.196681 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64fa570c-ecb7-4389-8e76-c6ea3dac319f admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:49:47.207419 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64fa570c-ecb7-4389-8e76-c6ea3dac319f admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:49:47.263527 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a49636b6-405c-4c36-91e0-806289604f66 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:49:47.264128 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a49636b6-405c-4c36-91e0-806289604f66 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:49:47.264409 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a49636b6-405c-4c36-91e0-806289604f66 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:49:47.264711 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a49636b6-405c-4c36-91e0-806289604f66 admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:49:47.265353 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a49636b6-405c-4c36-91e0-806289604f66 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:49:47.267312 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a49636b6-405c-4c36-91e0-806289604f66 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:49:47.288921 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1435/2895] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:49:47 2022] POST /identity/v3/users => generated 292 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:49:47.293389 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fc6baf4-36a9-495d-85ec-2a7ef3b05a3e admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:49:47.305053 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fc6baf4-36a9-495d-85ec-2a7ef3b05a3e admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:49:47.309716 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd52bd4c-816a-499e-a810-b897aa87647c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:49:47.310318 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd52bd4c-816a-499e-a810-b897aa87647c admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:49:47.310599 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd52bd4c-816a-499e-a810-b897aa87647c admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:49:47.310869 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd52bd4c-816a-499e-a810-b897aa87647c admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:49:47.311506 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd52bd4c-816a-499e-a810-b897aa87647c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:49:47.313253 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd52bd4c-816a-499e-a810-b897aa87647c admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:49:47.323687 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1461/2896] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:49:47 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:49:47.327718 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a49636b6-405c-4c36-91e0-806289604f66 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:49:47.338547 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a49636b6-405c-4c36-91e0-806289604f66 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:49:47.343442 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48316bb8-b4a0-45b7-ad80-0ac63a7c6ceb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:49:47.344040 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48316bb8-b4a0-45b7-ad80-0ac63a7c6ceb admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:49:47.344322 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48316bb8-b4a0-45b7-ad80-0ac63a7c6ceb admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:49:47.344594 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48316bb8-b4a0-45b7-ad80-0ac63a7c6ceb admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:49:47.345266 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48316bb8-b4a0-45b7-ad80-0ac63a7c6ceb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:49:47.346930 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48316bb8-b4a0-45b7-ad80-0ac63a7c6ceb admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:49:47.357227 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1436/2897] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:49:47 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:49:47.361388 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd52bd4c-816a-499e-a810-b897aa87647c admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:49:47.377598 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd52bd4c-816a-499e-a810-b897aa87647c admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:49:47.382620 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20ae7cef-6861-403a-ae40-a9db016fa62c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:49:47.383288 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20ae7cef-6861-403a-ae40-a9db016fa62c admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:49:47.383570 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20ae7cef-6861-403a-ae40-a9db016fa62c admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:49:47.383842 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20ae7cef-6861-403a-ae40-a9db016fa62c admin admin] PATH_INFO: `/v3/projects/216f629419cb42d08f7528b70020b99b/users/08c0cbcabba449f8bc716e64be77130a/roles/8f6cc574056c4278bc9778f14dc4b411` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:49:47.399873 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20ae7cef-6861-403a-ae40-a9db016fa62c admin admin] RBAC: Authorizing `identity:create_grant(project_id=216f629419cb42d08f7528b70020b99b, user_id=08c0cbcabba449f8bc716e64be77130a, role_id=8f6cc574056c4278bc9778f14dc4b411)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:49:47.402534 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20ae7cef-6861-403a-ae40-a9db016fa62c admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:49:47.415845 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1462/2898] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:49:47 2022] PUT /identity/v3/projects/216f629419cb42d08f7528b70020b99b/users/08c0cbcabba449f8bc716e64be77130a/roles/8f6cc574056c4278bc9778f14dc4b411 => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:49:47.481995 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07d8018f-e70c-4a1a-97d6-097dc238aabc None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:49:47.482212 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07d8018f-e70c-4a1a-97d6-097dc238aabc None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:49:47.482486 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07d8018f-e70c-4a1a-97d6-097dc238aabc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:49:47.511347 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-07d8018f-e70c-4a1a-97d6-097dc238aabc None None] MFA Rules not processed for user `08c0cbcabba449f8bc716e64be77130a`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:49:47.512539 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-07d8018f-e70c-4a1a-97d6-097dc238aabc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:49:47.521620 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:49:47.576763 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1437/2899] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:49:47 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:49:47.588624 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20ae7cef-6861-403a-ae40-a9db016fa62c admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:49:47.590663 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-20ae7cef-6861-403a-ae40-a9db016fa62c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:49:47.647589 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20ae7cef-6861-403a-ae40-a9db016fa62c admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:49:47.651290 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6081244c-a252-4af6-862c-c731a8043c16 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ffbc136c77b84e8c87128f7247f4ec03', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:49:47.651894 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6081244c-a252-4af6-862c-c731a8043c16 service cinder] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:49:47.652178 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6081244c-a252-4af6-862c-c731a8043c16 service cinder] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:49:47.652452 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6081244c-a252-4af6-862c-c731a8043c16 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:49:47.661403 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6081244c-a252-4af6-862c-c731a8043c16 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:49:47.662838 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6081244c-a252-4af6-862c-c731a8043c16 service cinder] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:49:47.671626 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1463/2900] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:49:47 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:49:50.238500 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07d8018f-e70c-4a1a-97d6-097dc238aabc None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:49:50.299962 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07d8018f-e70c-4a1a-97d6-097dc238aabc None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:49:50.304484 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-092e1ced-2933-4b2f-bafa-350c6f21cd71 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:49:50.305131 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-092e1ced-2933-4b2f-bafa-350c6f21cd71 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:49:50.305408 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-092e1ced-2933-4b2f-bafa-350c6f21cd71 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:49:50.305678 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-092e1ced-2933-4b2f-bafa-350c6f21cd71 admin admin] PATH_INFO: `/v3/users/08c0cbcabba449f8bc716e64be77130a` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:49:50.306787 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-092e1ced-2933-4b2f-bafa-350c6f21cd71 admin admin] RBAC: Authorizing `identity:delete_user(user_id=08c0cbcabba449f8bc716e64be77130a)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:49:50.308710 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-092e1ced-2933-4b2f-bafa-350c6f21cd71 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:49:50.354168 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-092e1ced-2933-4b2f-bafa-350c6f21cd71 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '08c0cbcabba449f8bc716e64be77130a'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:49:50.362290 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-092e1ced-2933-4b2f-bafa-350c6f21cd71 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '08c0cbcabba449f8bc716e64be77130a'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:49:50.374794 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-092e1ced-2933-4b2f-bafa-350c6f21cd71 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '08c0cbcabba449f8bc716e64be77130a'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:49:50.375460 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-092e1ced-2933-4b2f-bafa-350c6f21cd71 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '08c0cbcabba449f8bc716e64be77130a'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:49:50.383120 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-092e1ced-2933-4b2f-bafa-350c6f21cd71 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '08c0cbcabba449f8bc716e64be77130a'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:49:50.389038 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1438/2901] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:49:50 2022] DELETE /identity/v3/users/08c0cbcabba449f8bc716e64be77130a => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 04:49:50.394409 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8abc73d1-5c37-44eb-b992-71ac1fa31807 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:49:50.394910 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8abc73d1-5c37-44eb-b992-71ac1fa31807 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:49:50.395038 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8abc73d1-5c37-44eb-b992-71ac1fa31807 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:49:50.414225 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-8abc73d1-5c37-44eb-b992-71ac1fa31807 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:49:50.415422 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-8abc73d1-5c37-44eb-b992-71ac1fa31807 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:49:50.474098 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1464/2902] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:49:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 04:49:50.483761 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-092e1ced-2933-4b2f-bafa-350c6f21cd71 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:49:50.485205 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-092e1ced-2933-4b2f-bafa-350c6f21cd71 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:49:50.520312 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:49:50.545407 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-092e1ced-2933-4b2f-bafa-350c6f21cd71 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:49:50.549338 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a131fbc-a5dd-4868-ab33-aea83c68e844 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:49:50.549936 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a131fbc-a5dd-4868-ab33-aea83c68e844 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:49:50.550214 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a131fbc-a5dd-4868-ab33-aea83c68e844 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:49:50.550485 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a131fbc-a5dd-4868-ab33-aea83c68e844 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:49:50.559343 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a131fbc-a5dd-4868-ab33-aea83c68e844 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:49:50.560783 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a131fbc-a5dd-4868-ab33-aea83c68e844 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:49:50.570552 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1439/2903] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:49:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:49:50.834316 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8abc73d1-5c37-44eb-b992-71ac1fa31807 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:49:50.835875 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-8abc73d1-5c37-44eb-b992-71ac1fa31807 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:49:50.849610 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8abc73d1-5c37-44eb-b992-71ac1fa31807 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:49:50.854153 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a683c456-7781-494d-9c9b-9c3c59c0beb8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:49:50.854795 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a683c456-7781-494d-9c9b-9c3c59c0beb8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:49:50.855074 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a683c456-7781-494d-9c9b-9c3c59c0beb8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:49:50.855343 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a683c456-7781-494d-9c9b-9c3c59c0beb8 admin admin] PATH_INFO: `/v3/projects/216f629419cb42d08f7528b70020b99b` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:49:50.856496 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a683c456-7781-494d-9c9b-9c3c59c0beb8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=216f629419cb42d08f7528b70020b99b)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:49:50.858484 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a683c456-7781-494d-9c9b-9c3c59c0beb8 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:49:50.892484 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=96,keystone:INSERT=2,keystone:DELETE=2 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:49:50.906837 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-a683c456-7781-494d-9c9b-9c3c59c0beb8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '216f629419cb42d08f7528b70020b99b', 'request_id': 'req-a683c456-7781-494d-9c9b-9c3c59c0beb8'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:49:50.914670 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-a683c456-7781-494d-9c9b-9c3c59c0beb8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '216f629419cb42d08f7528b70020b99b', 'request_id': 'req-a683c456-7781-494d-9c9b-9c3c59c0beb8'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:49:50.920323 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-a683c456-7781-494d-9c9b-9c3c59c0beb8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '216f629419cb42d08f7528b70020b99b', 'request_id': 'req-a683c456-7781-494d-9c9b-9c3c59c0beb8'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:49:50.932241 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-a683c456-7781-494d-9c9b-9c3c59c0beb8 admin admin] The token cache is being invalidate because project 216f629419cb42d08f7528b70020b99b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81434) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:49:50.932532 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-a683c456-7781-494d-9c9b-9c3c59c0beb8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:49:50.933190 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-a683c456-7781-494d-9c9b-9c3c59c0beb8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:49:50.934874 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1465/2904] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:49:50 2022] DELETE /identity/v3/projects/216f629419cb42d08f7528b70020b99b => generated 0 bytes in 101 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:49:50.939919 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afc0cd23-623b-4c09-9846-7fb6c54ef268 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:49:50.940202 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afc0cd23-623b-4c09-9846-7fb6c54ef268 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:49:50.940475 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afc0cd23-623b-4c09-9846-7fb6c54ef268 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:49:50.959284 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-afc0cd23-623b-4c09-9846-7fb6c54ef268 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:49:50.960476 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-afc0cd23-623b-4c09-9846-7fb6c54ef268 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:49:51.019700 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1440/2905] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:49:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:49:51.027335 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f38a822-e685-4c4d-9b11-98935d71e808 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:49:51.027628 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f38a822-e685-4c4d-9b11-98935d71e808 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:49:51.027906 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f38a822-e685-4c4d-9b11-98935d71e808 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:49:51.048199 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-8f38a822-e685-4c4d-9b11-98935d71e808 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:49:51.049985 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-8f38a822-e685-4c4d-9b11-98935d71e808 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:49:51.057819 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1466/2906] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:49:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:49:51.061987 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afc0cd23-623b-4c09-9846-7fb6c54ef268 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:49:51.071681 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afc0cd23-623b-4c09-9846-7fb6c54ef268 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:49:51.076178 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f09642c-40eb-4c32-9733-5addabda41c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:49:51.076794 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f09642c-40eb-4c32-9733-5addabda41c3 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:49:51.077070 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f09642c-40eb-4c32-9733-5addabda41c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:49:51.077336 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f09642c-40eb-4c32-9733-5addabda41c3 admin admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:49:51.077968 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f09642c-40eb-4c32-9733-5addabda41c3 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:49:51.078265 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f09642c-40eb-4c32-9733-5addabda41c3 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:49:51.079863 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f09642c-40eb-4c32-9733-5addabda41c3 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:49:51.091755 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1441/2907] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Sep 25 04:49:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:49:51.096818 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f38a822-e685-4c4d-9b11-98935d71e808 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:49:51.105963 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f38a822-e685-4c4d-9b11-98935d71e808 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:49:51.110635 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e73e0986-0ec7-43e8-acda-c2d4c83a960c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:49:51.111842 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e73e0986-0ec7-43e8-acda-c2d4c83a960c admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:49:51.112797 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e73e0986-0ec7-43e8-acda-c2d4c83a960c admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:49:51.113080 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e73e0986-0ec7-43e8-acda-c2d4c83a960c admin admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:49:51.113714 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e73e0986-0ec7-43e8-acda-c2d4c83a960c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:49:51.115601 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e73e0986-0ec7-43e8-acda-c2d4c83a960c admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:49:51.144101 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1467/2908] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:49:51 2022] POST /identity/v3/projects => generated 373 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:49:51.148408 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f09642c-40eb-4c32-9733-5addabda41c3 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:49:51.159002 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f09642c-40eb-4c32-9733-5addabda41c3 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:49:51.215740 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32dcf48a-3f05-49c5-9d9c-e18139b1230f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:49:51.216341 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32dcf48a-3f05-49c5-9d9c-e18139b1230f admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:49:51.216619 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32dcf48a-3f05-49c5-9d9c-e18139b1230f admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:49:51.216918 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32dcf48a-3f05-49c5-9d9c-e18139b1230f admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:49:51.217545 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32dcf48a-3f05-49c5-9d9c-e18139b1230f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:49:51.219444 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32dcf48a-3f05-49c5-9d9c-e18139b1230f admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:49:51.241305 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1442/2909] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:49:51 2022] POST /identity/v3/users => generated 287 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:49:51.245623 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e73e0986-0ec7-43e8-acda-c2d4c83a960c admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:49:51.256047 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e73e0986-0ec7-43e8-acda-c2d4c83a960c admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:49:51.260587 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b453c9c-b2cd-4966-9e13-e384f712c84e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:49:51.261209 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b453c9c-b2cd-4966-9e13-e384f712c84e admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:49:51.261523 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b453c9c-b2cd-4966-9e13-e384f712c84e admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:49:51.261795 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b453c9c-b2cd-4966-9e13-e384f712c84e admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:49:51.262432 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b453c9c-b2cd-4966-9e13-e384f712c84e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:49:51.264017 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b453c9c-b2cd-4966-9e13-e384f712c84e admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:49:51.274509 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1468/2910] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:49:51 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:49:51.278435 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32dcf48a-3f05-49c5-9d9c-e18139b1230f admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:49:51.289361 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32dcf48a-3f05-49c5-9d9c-e18139b1230f admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:49:51.294028 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a400d61-f75a-4228-8483-57c37c6d6599 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:49:51.294638 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a400d61-f75a-4228-8483-57c37c6d6599 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:49:51.294904 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a400d61-f75a-4228-8483-57c37c6d6599 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:49:51.295177 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a400d61-f75a-4228-8483-57c37c6d6599 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:49:51.295862 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a400d61-f75a-4228-8483-57c37c6d6599 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:49:51.297798 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a400d61-f75a-4228-8483-57c37c6d6599 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:49:51.308231 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1443/2911] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:49:51 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:49:51.312394 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b453c9c-b2cd-4966-9e13-e384f712c84e admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:49:51.322508 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b453c9c-b2cd-4966-9e13-e384f712c84e admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:49:51.327103 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2b30372-f6f2-4e48-b0a7-86d47a24318a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:49:51.327743 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2b30372-f6f2-4e48-b0a7-86d47a24318a admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:49:51.328020 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2b30372-f6f2-4e48-b0a7-86d47a24318a admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:49:51.328291 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2b30372-f6f2-4e48-b0a7-86d47a24318a admin admin] PATH_INFO: `/v3/projects/aef4cf6e9fed425881c0b6c02d3aae95/users/b5be5334d0094fa9bf30a6bc5a961f25/roles/8f6cc574056c4278bc9778f14dc4b411` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:49:51.343620 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2b30372-f6f2-4e48-b0a7-86d47a24318a admin admin] RBAC: Authorizing `identity:create_grant(project_id=aef4cf6e9fed425881c0b6c02d3aae95, user_id=b5be5334d0094fa9bf30a6bc5a961f25, role_id=8f6cc574056c4278bc9778f14dc4b411)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:49:51.347112 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2b30372-f6f2-4e48-b0a7-86d47a24318a admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:49:51.359508 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1469/2912] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:49:51 2022] PUT /identity/v3/projects/aef4cf6e9fed425881c0b6c02d3aae95/users/b5be5334d0094fa9bf30a6bc5a961f25/roles/8f6cc574056c4278bc9778f14dc4b411 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:49:51.373256 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-281e7731-6100-462e-a172-8b07b996103a None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:49:51.373554 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-281e7731-6100-462e-a172-8b07b996103a None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:49:51.373818 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-281e7731-6100-462e-a172-8b07b996103a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:49:51.401371 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-281e7731-6100-462e-a172-8b07b996103a None None] MFA Rules not processed for user `b5be5334d0094fa9bf30a6bc5a961f25`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:49:51.402572 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-281e7731-6100-462e-a172-8b07b996103a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:49:51.431722 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:49:51.468173 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1444/2913] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:49:51 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:49:51.480375 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2b30372-f6f2-4e48-b0a7-86d47a24318a admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:49:51.482123 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-e2b30372-f6f2-4e48-b0a7-86d47a24318a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:49:51.538535 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2b30372-f6f2-4e48-b0a7-86d47a24318a admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:49:51.542100 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3520dc14-07d4-4cf5-91ab-ed5f7393ed19 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ffbc136c77b84e8c87128f7247f4ec03', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:49:51.542703 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3520dc14-07d4-4cf5-91ab-ed5f7393ed19 service cinder] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:49:51.542986 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3520dc14-07d4-4cf5-91ab-ed5f7393ed19 service cinder] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:49:51.543259 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3520dc14-07d4-4cf5-91ab-ed5f7393ed19 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:49:51.551610 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3520dc14-07d4-4cf5-91ab-ed5f7393ed19 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:49:51.553076 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3520dc14-07d4-4cf5-91ab-ed5f7393ed19 service cinder] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:49:51.561822 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1470/2914] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:49:51 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 82 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:49:51.586848 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-281e7731-6100-462e-a172-8b07b996103a None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:49:51.647847 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-281e7731-6100-462e-a172-8b07b996103a None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:49:51.652385 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e0c1a22-e551-4e2f-9750-8af134821998 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:49:51.653032 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e0c1a22-e551-4e2f-9750-8af134821998 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:49:51.653310 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e0c1a22-e551-4e2f-9750-8af134821998 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:49:51.653581 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e0c1a22-e551-4e2f-9750-8af134821998 admin admin] PATH_INFO: `/v3/users/b5be5334d0094fa9bf30a6bc5a961f25` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:49:51.654693 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e0c1a22-e551-4e2f-9750-8af134821998 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b5be5334d0094fa9bf30a6bc5a961f25)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:49:51.656593 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e0c1a22-e551-4e2f-9750-8af134821998 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:49:51.701229 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-4e0c1a22-e551-4e2f-9750-8af134821998 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b5be5334d0094fa9bf30a6bc5a961f25'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:49:51.708955 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-4e0c1a22-e551-4e2f-9750-8af134821998 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b5be5334d0094fa9bf30a6bc5a961f25'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:49:51.721199 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-4e0c1a22-e551-4e2f-9750-8af134821998 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b5be5334d0094fa9bf30a6bc5a961f25'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:49:51.721846 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-4e0c1a22-e551-4e2f-9750-8af134821998 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b5be5334d0094fa9bf30a6bc5a961f25'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:49:51.728551 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-4e0c1a22-e551-4e2f-9750-8af134821998 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b5be5334d0094fa9bf30a6bc5a961f25'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:49:51.734315 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1445/2915] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:49:51 2022] DELETE /identity/v3/users/b5be5334d0094fa9bf30a6bc5a961f25 => generated 0 bytes in 148 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:49:51.738846 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92654f21-9442-48b9-a868-1b3d29feba97 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:49:51.739138 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92654f21-9442-48b9-a868-1b3d29feba97 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:49:51.739417 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92654f21-9442-48b9-a868-1b3d29feba97 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:49:51.758950 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-92654f21-9442-48b9-a868-1b3d29feba97 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:49:51.760154 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-92654f21-9442-48b9-a868-1b3d29feba97 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:49:51.809587 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=92,keystone:DELETE=5,keystone:INSERT=3 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:49:51.828230 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1471/2916] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:49:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:49:51.839310 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e0c1a22-e551-4e2f-9750-8af134821998 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:49:51.840872 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-4e0c1a22-e551-4e2f-9750-8af134821998 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:49:51.899257 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:49:51.903476 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e0c1a22-e551-4e2f-9750-8af134821998 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:49:51.908201 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d370b55-49e0-4493-bf92-74e18d8ac878 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:49:51.909113 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d370b55-49e0-4493-bf92-74e18d8ac878 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:49:51.909770 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d370b55-49e0-4493-bf92-74e18d8ac878 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:49:51.910154 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d370b55-49e0-4493-bf92-74e18d8ac878 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:49:51.919734 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d370b55-49e0-4493-bf92-74e18d8ac878 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:49:51.921236 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d370b55-49e0-4493-bf92-74e18d8ac878 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:49:51.931067 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1446/2917] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:49:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:49:52.218447 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92654f21-9442-48b9-a868-1b3d29feba97 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:49:52.219942 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-92654f21-9442-48b9-a868-1b3d29feba97 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:49:52.235307 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92654f21-9442-48b9-a868-1b3d29feba97 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:49:52.241787 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92da0ddc-124f-4bfc-93e6-cb3bae57f6ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:49:52.242434 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92da0ddc-124f-4bfc-93e6-cb3bae57f6ed admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:49:52.242714 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92da0ddc-124f-4bfc-93e6-cb3bae57f6ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:49:52.242996 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92da0ddc-124f-4bfc-93e6-cb3bae57f6ed admin admin] PATH_INFO: `/v3/projects/aef4cf6e9fed425881c0b6c02d3aae95` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:49:52.244246 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92da0ddc-124f-4bfc-93e6-cb3bae57f6ed admin admin] RBAC: Authorizing `identity:delete_project(project_id=aef4cf6e9fed425881c0b6c02d3aae95)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:49:52.246537 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92da0ddc-124f-4bfc-93e6-cb3bae57f6ed admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:49:52.299376 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-92da0ddc-124f-4bfc-93e6-cb3bae57f6ed admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'aef4cf6e9fed425881c0b6c02d3aae95', 'request_id': 'req-92da0ddc-124f-4bfc-93e6-cb3bae57f6ed'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:49:52.309187 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-92da0ddc-124f-4bfc-93e6-cb3bae57f6ed admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'aef4cf6e9fed425881c0b6c02d3aae95', 'request_id': 'req-92da0ddc-124f-4bfc-93e6-cb3bae57f6ed'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:49:52.317479 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-92da0ddc-124f-4bfc-93e6-cb3bae57f6ed admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'aef4cf6e9fed425881c0b6c02d3aae95', 'request_id': 'req-92da0ddc-124f-4bfc-93e6-cb3bae57f6ed'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:49:52.330811 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-92da0ddc-124f-4bfc-93e6-cb3bae57f6ed admin admin] The token cache is being invalidate because project aef4cf6e9fed425881c0b6c02d3aae95 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81434) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:49:52.331026 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-92da0ddc-124f-4bfc-93e6-cb3bae57f6ed admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:49:52.331789 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-92da0ddc-124f-4bfc-93e6-cb3bae57f6ed admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:49:52.333696 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1472/2918] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:49:52 2022] DELETE /identity/v3/projects/aef4cf6e9fed425881c0b6c02d3aae95 => generated 0 bytes in 116 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 04:49:52.338609 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-508ffd5a-6e1b-43ac-a9ad-446961cc33f4 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:49:52.339480 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-508ffd5a-6e1b-43ac-a9ad-446961cc33f4 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:49:52.339480 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-508ffd5a-6e1b-43ac-a9ad-446961cc33f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:49:52.362900 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-508ffd5a-6e1b-43ac-a9ad-446961cc33f4 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:49:52.364127 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-508ffd5a-6e1b-43ac-a9ad-446961cc33f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:49:52.441537 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1447/2919] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:49:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 25 04:49:52.450595 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5010e2ee-e42e-4782-ab4d-f0818ffdd6c3 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:49:52.450886 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5010e2ee-e42e-4782-ab4d-f0818ffdd6c3 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:49:52.451164 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5010e2ee-e42e-4782-ab4d-f0818ffdd6c3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:49:52.477530 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-5010e2ee-e42e-4782-ab4d-f0818ffdd6c3 None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:49:52.478783 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-5010e2ee-e42e-4782-ab4d-f0818ffdd6c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:49:52.492542 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1473/2920] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:49:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 43 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:49:52.497786 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-508ffd5a-6e1b-43ac-a9ad-446961cc33f4 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:49:52.508596 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-508ffd5a-6e1b-43ac-a9ad-446961cc33f4 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:49:52.517657 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-749b84d4-2a7c-40c9-8b26-4887044be324 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:49:52.518430 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-749b84d4-2a7c-40c9-8b26-4887044be324 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:49:52.518713 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-749b84d4-2a7c-40c9-8b26-4887044be324 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:49:52.518988 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-749b84d4-2a7c-40c9-8b26-4887044be324 admin admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:49:52.519668 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-749b84d4-2a7c-40c9-8b26-4887044be324 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 25 04:49:52.519971 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-749b84d4-2a7c-40c9-8b26-4887044be324 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:49:52.521638 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-749b84d4-2a7c-40c9-8b26-4887044be324 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:49:52.536970 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1448/2921] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Sep 25 04:49:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 25 04:49:52.541382 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5010e2ee-e42e-4782-ab4d-f0818ffdd6c3 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:49:52.551193 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5010e2ee-e42e-4782-ab4d-f0818ffdd6c3 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:49:52.556327 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88d1fd90-97bd-455e-ad30-809dfbb56233 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:49:52.556973 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88d1fd90-97bd-455e-ad30-809dfbb56233 admin admin] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:49:52.557250 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88d1fd90-97bd-455e-ad30-809dfbb56233 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:49:52.557519 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88d1fd90-97bd-455e-ad30-809dfbb56233 admin admin] PATH_INFO: `/v3/projects` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:49:52.558153 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88d1fd90-97bd-455e-ad30-809dfbb56233 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:49:52.560399 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88d1fd90-97bd-455e-ad30-809dfbb56233 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:49:52.592318 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1474/2922] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Sep 25 04:49:52 2022] POST /identity/v3/projects => generated 383 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:49:52.598053 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-749b84d4-2a7c-40c9-8b26-4887044be324 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:49:52.609130 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-749b84d4-2a7c-40c9-8b26-4887044be324 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:49:52.675644 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-167e7466-5be6-4033-8471-bf20c2ff0d5e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:49:52.676258 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-167e7466-5be6-4033-8471-bf20c2ff0d5e admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:49:52.676530 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-167e7466-5be6-4033-8471-bf20c2ff0d5e admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:49:52.676836 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-167e7466-5be6-4033-8471-bf20c2ff0d5e admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:49:52.677469 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-167e7466-5be6-4033-8471-bf20c2ff0d5e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:49:52.679400 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-167e7466-5be6-4033-8471-bf20c2ff0d5e admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:49:52.703119 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1449/2923] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Sep 25 04:49:52 2022] POST /identity/v3/users => generated 292 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 25 04:49:52.708280 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88d1fd90-97bd-455e-ad30-809dfbb56233 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:49:52.721737 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88d1fd90-97bd-455e-ad30-809dfbb56233 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:49:52.726736 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-154a1504-960a-4f4a-9a5b-19f99e990eef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:49:52.727419 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-154a1504-960a-4f4a-9a5b-19f99e990eef admin admin] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:49:52.727698 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-154a1504-960a-4f4a-9a5b-19f99e990eef admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:49:52.727970 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-154a1504-960a-4f4a-9a5b-19f99e990eef admin admin] PATH_INFO: `/v3/roles` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:49:52.728621 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-154a1504-960a-4f4a-9a5b-19f99e990eef admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:49:52.731245 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-154a1504-960a-4f4a-9a5b-19f99e990eef admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:49:52.742687 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1475/2924] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:49:52 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 25 04:49:52.747090 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-167e7466-5be6-4033-8471-bf20c2ff0d5e admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:49:52.758205 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-167e7466-5be6-4033-8471-bf20c2ff0d5e admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:49:52.763040 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7ef2a63-57f9-48b5-b239-651f3ef87cf8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:49:52.763648 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7ef2a63-57f9-48b5-b239-651f3ef87cf8 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:49:52.763933 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7ef2a63-57f9-48b5-b239-651f3ef87cf8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:49:52.764211 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7ef2a63-57f9-48b5-b239-651f3ef87cf8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:49:52.764895 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7ef2a63-57f9-48b5-b239-651f3ef87cf8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:49:52.766505 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7ef2a63-57f9-48b5-b239-651f3ef87cf8 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:49:52.779010 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1450/2925] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Sep 25 04:49:52 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 25 04:49:52.784346 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-154a1504-960a-4f4a-9a5b-19f99e990eef admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:49:52.795819 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-154a1504-960a-4f4a-9a5b-19f99e990eef admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:49:52.802325 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acae7633-abd7-42a1-8dcf-c8c0d3f7546a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:49:52.802980 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acae7633-abd7-42a1-8dcf-c8c0d3f7546a admin admin] REQUEST_METHOD: `PUT` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:49:52.803258 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acae7633-abd7-42a1-8dcf-c8c0d3f7546a admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:49:52.803532 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acae7633-abd7-42a1-8dcf-c8c0d3f7546a admin admin] PATH_INFO: `/v3/projects/38de016089e0474d9a2b16cfecaf0a60/users/0c481f7a811a438e809b3bb8a8e4c127/roles/8f6cc574056c4278bc9778f14dc4b411` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:49:52.823244 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acae7633-abd7-42a1-8dcf-c8c0d3f7546a admin admin] RBAC: Authorizing `identity:create_grant(project_id=38de016089e0474d9a2b16cfecaf0a60, user_id=0c481f7a811a438e809b3bb8a8e4c127, role_id=8f6cc574056c4278bc9778f14dc4b411)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:49:52.826032 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acae7633-abd7-42a1-8dcf-c8c0d3f7546a admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:49:52.843061 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1476/2926] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Sep 25 04:49:52 2022] PUT /identity/v3/projects/38de016089e0474d9a2b16cfecaf0a60/users/0c481f7a811a438e809b3bb8a8e4c127/roles/8f6cc574056c4278bc9778f14dc4b411 => generated 0 bytes in 60 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:49:52.848358 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bda9d5c-01c3-45cd-afcb-58cb747bfe5f None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:49:52.848666 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bda9d5c-01c3-45cd-afcb-58cb747bfe5f None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:49:52.848961 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bda9d5c-01c3-45cd-afcb-58cb747bfe5f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:49:52.871197 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-2bda9d5c-01c3-45cd-afcb-58cb747bfe5f None None] MFA Rules not processed for user `e61aa4e8d68f455e96668bc70e0a6679`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:49:52.872429 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-2bda9d5c-01c3-45cd-afcb-58cb747bfe5f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:49:52.925316 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:49:52.935778 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1451/2927] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:49:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:49:52.946219 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acae7633-abd7-42a1-8dcf-c8c0d3f7546a admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:49:52.947926 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-acae7633-abd7-42a1-8dcf-c8c0d3f7546a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:49:53.005841 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acae7633-abd7-42a1-8dcf-c8c0d3f7546a admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:49:53.010025 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24d0b265-82ca-41ad-a0c1-4c953f659d0d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:49:53.010702 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d0b265-82ca-41ad-a0c1-4c953f659d0d service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:49:53.010979 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d0b265-82ca-41ad-a0c1-4c953f659d0d service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:49:53.011275 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d0b265-82ca-41ad-a0c1-4c953f659d0d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:49:53.024075 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24d0b265-82ca-41ad-a0c1-4c953f659d0d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:49:53.025533 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24d0b265-82ca-41ad-a0c1-4c953f659d0d service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:49:53.035294 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1477/2928] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:49:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:49:53.682363 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52d80e8c-552d-4c0c-ac98-275cc290042c None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:49:53.682985 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52d80e8c-552d-4c0c-ac98-275cc290042c None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:49:53.682985 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52d80e8c-552d-4c0c-ac98-275cc290042c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:49:53.713130 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-52d80e8c-552d-4c0c-ac98-275cc290042c None None] MFA Rules not processed for user `0c481f7a811a438e809b3bb8a8e4c127`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:49:53.714688 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-52d80e8c-552d-4c0c-ac98-275cc290042c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:49:53.781239 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1452/2929] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:49:53 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:49:53.792133 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24d0b265-82ca-41ad-a0c1-4c953f659d0d service neutron] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:49:53.793677 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-24d0b265-82ca-41ad-a0c1-4c953f659d0d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:49:53.817793 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:49:53.855197 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24d0b265-82ca-41ad-a0c1-4c953f659d0d service neutron] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:49:53.859576 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b7ddc43-5987-497f-a941-bbb3d5a14156 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ffbc136c77b84e8c87128f7247f4ec03', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:49:53.860127 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b7ddc43-5987-497f-a941-bbb3d5a14156 service cinder] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:49:53.860418 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b7ddc43-5987-497f-a941-bbb3d5a14156 service cinder] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:49:53.860729 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b7ddc43-5987-497f-a941-bbb3d5a14156 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:49:53.870480 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b7ddc43-5987-497f-a941-bbb3d5a14156 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:49:53.871967 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b7ddc43-5987-497f-a941-bbb3d5a14156 service cinder] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:49:53.881007 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1478/2930] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:49:53 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:49:55.235285 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab33a168-b98e-4fd7-9594-87122f77e9b8 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:49:55.236323 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab33a168-b98e-4fd7-9594-87122f77e9b8 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:49:55.236323 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab33a168-b98e-4fd7-9594-87122f77e9b8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:49:55.255981 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-ab33a168-b98e-4fd7-9594-87122f77e9b8 None None] MFA Rules not processed for user `0c481f7a811a438e809b3bb8a8e4c127`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:49:55.257214 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-ab33a168-b98e-4fd7-9594-87122f77e9b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:49:55.263983 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1453/2931] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:49:55 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:50:03.877246 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=17 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:50:05.249526 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=36 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:50:59.961658 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b7ddc43-5987-497f-a941-bbb3d5a14156 service cinder] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:50:59.971417 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b7ddc43-5987-497f-a941-bbb3d5a14156 service cinder] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:50:59.975064 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e57b0b98-1333-4bb1-92ba-69f64f30a0a5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:50:59.975680 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e57b0b98-1333-4bb1-92ba-69f64f30a0a5 service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:50:59.975963 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e57b0b98-1333-4bb1-92ba-69f64f30a0a5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:50:59.976234 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e57b0b98-1333-4bb1-92ba-69f64f30a0a5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:50:59.977957 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-e57b0b98-1333-4bb1-92ba-69f64f30a0a5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:51:00.048965 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e57b0b98-1333-4bb1-92ba-69f64f30a0a5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:51:00.050440 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e57b0b98-1333-4bb1-92ba-69f64f30a0a5 service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:51:00.063599 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1479/2932] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:50:59 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:51:01.259825 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab33a168-b98e-4fd7-9594-87122f77e9b8 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:51:01.270666 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab33a168-b98e-4fd7-9594-87122f77e9b8 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:51:01.274446 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa90c2a0-5032-4913-9eb9-ea5c4ed00bcb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:51:01.275056 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa90c2a0-5032-4913-9eb9-ea5c4ed00bcb service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:51:01.275359 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa90c2a0-5032-4913-9eb9-ea5c4ed00bcb service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:51:01.275627 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa90c2a0-5032-4913-9eb9-ea5c4ed00bcb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:51:01.277378 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-aa90c2a0-5032-4913-9eb9-ea5c4ed00bcb service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:51:01.291394 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa90c2a0-5032-4913-9eb9-ea5c4ed00bcb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:51:01.292948 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa90c2a0-5032-4913-9eb9-ea5c4ed00bcb service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:51:01.304925 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1454/2933] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:51:01 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:51:10.057097 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=34 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:51:11.300140 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=6 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:51:12.361461 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfacb219-616f-465b-80c0-784c211735e5 None None] REQUEST_METHOD: `POST` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:51:12.362009 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfacb219-616f-465b-80c0-784c211735e5 None None] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:51:12.362125 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfacb219-616f-465b-80c0-784c211735e5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:51:12.382767 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.auth.core [None req-cfacb219-616f-465b-80c0-784c211735e5 None None] MFA Rules not processed for user `0c481f7a811a438e809b3bb8a8e4c127`. Rule list: `[]` (Enabled: `True`). {{(pid=81434) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 25 04:51:12.383971 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-cfacb219-616f-465b-80c0-784c211735e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:51:12.392053 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1480/2934] 10.222.0.25 () {58 vars in 958 bytes} [Sun Sep 25 04:51:12 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 25 04:51:12.627401 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa90c2a0-5032-4913-9eb9-ea5c4ed00bcb service neutron] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:51:12.636079 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa90c2a0-5032-4913-9eb9-ea5c4ed00bcb service neutron] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:51:12.640606 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9c4fec7-2b91-4bc5-8f32-fc62bf879dd3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:51:12.641260 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9c4fec7-2b91-4bc5-8f32-fc62bf879dd3 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:51:12.641538 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9c4fec7-2b91-4bc5-8f32-fc62bf879dd3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:51:12.641815 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9c4fec7-2b91-4bc5-8f32-fc62bf879dd3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:51:12.643615 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-d9c4fec7-2b91-4bc5-8f32-fc62bf879dd3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:51:12.653358 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9c4fec7-2b91-4bc5-8f32-fc62bf879dd3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:51:12.654805 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9c4fec7-2b91-4bc5-8f32-fc62bf879dd3 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:51:12.663139 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1455/2935] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:51:12 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:51:22.375895 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=3 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:51:22.659243 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=6 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:52:05.417623 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfacb219-616f-465b-80c0-784c211735e5 None None] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:52:05.427592 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfacb219-616f-465b-80c0-784c211735e5 None None] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:52:05.431199 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5117a8da-16bb-4a99-9fdc-1723b2c95f01 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:52:05.431793 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5117a8da-16bb-4a99-9fdc-1723b2c95f01 service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:52:05.432068 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5117a8da-16bb-4a99-9fdc-1723b2c95f01 service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:52:05.432338 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5117a8da-16bb-4a99-9fdc-1723b2c95f01 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:52:05.434059 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-5117a8da-16bb-4a99-9fdc-1723b2c95f01 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:52:05.444405 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5117a8da-16bb-4a99-9fdc-1723b2c95f01 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:52:05.445853 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5117a8da-16bb-4a99-9fdc-1723b2c95f01 service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:52:05.454442 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1481/2936] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:52:05 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:52:14.104260 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9c4fec7-2b91-4bc5-8f32-fc62bf879dd3 service neutron] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:52:14.105707 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-d9c4fec7-2b91-4bc5-8f32-fc62bf879dd3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:52:14.161148 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5117a8da-16bb-4a99-9fdc-1723b2c95f01 service neutron] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:52:14.174549 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5117a8da-16bb-4a99-9fdc-1723b2c95f01 service neutron] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:52:14.179760 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03ea5e43-372f-4dc8-9b80-971d5aaac7c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:52:14.180266 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03ea5e43-372f-4dc8-9b80-971d5aaac7c2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:52:14.180548 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03ea5e43-372f-4dc8-9b80-971d5aaac7c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:52:14.180844 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03ea5e43-372f-4dc8-9b80-971d5aaac7c2 admin admin] PATH_INFO: `/v3/users/0c481f7a811a438e809b3bb8a8e4c127` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:52:14.183866 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03ea5e43-372f-4dc8-9b80-971d5aaac7c2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0c481f7a811a438e809b3bb8a8e4c127)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:52:14.186281 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03ea5e43-372f-4dc8-9b80-971d5aaac7c2 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:52:14.186390 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9c4fec7-2b91-4bc5-8f32-fc62bf879dd3 service neutron] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:52:14.192265 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-098e856b-6026-4fc1-8afa-9e5a74c0ab12 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '676d04ef5a984edcaad63d9156568660', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:52:14.192931 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-098e856b-6026-4fc1-8afa-9e5a74c0ab12 service placement] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:52:14.193208 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-098e856b-6026-4fc1-8afa-9e5a74c0ab12 service placement] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:52:14.193483 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-098e856b-6026-4fc1-8afa-9e5a74c0ab12 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:52:14.195245 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-098e856b-6026-4fc1-8afa-9e5a74c0ab12 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:52:14.208066 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-098e856b-6026-4fc1-8afa-9e5a74c0ab12 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:52:14.209363 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-098e856b-6026-4fc1-8afa-9e5a74c0ab12 service placement] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:52:14.221246 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1456/2937] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:52:14 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 117 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:52:14.236611 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-03ea5e43-372f-4dc8-9b80-971d5aaac7c2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0c481f7a811a438e809b3bb8a8e4c127'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:52:14.245754 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-03ea5e43-372f-4dc8-9b80-971d5aaac7c2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0c481f7a811a438e809b3bb8a8e4c127'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:52:14.259624 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-03ea5e43-372f-4dc8-9b80-971d5aaac7c2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0c481f7a811a438e809b3bb8a8e4c127'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:52:14.260408 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-03ea5e43-372f-4dc8-9b80-971d5aaac7c2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0c481f7a811a438e809b3bb8a8e4c127'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:52:14.267611 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-03ea5e43-372f-4dc8-9b80-971d5aaac7c2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0c481f7a811a438e809b3bb8a8e4c127'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:52:14.273615 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1482/2938] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:52:14 2022] DELETE /identity/v3/users/0c481f7a811a438e809b3bb8a8e4c127 => generated 0 bytes in 113 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 25 04:52:14.446087 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-098e856b-6026-4fc1-8afa-9e5a74c0ab12 service placement] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:52:14.459435 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-098e856b-6026-4fc1-8afa-9e5a74c0ab12 service placement] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:52:14.519497 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f3fb26e-5eb3-42fe-b466-94bcff888d92 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:52:14.520123 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f3fb26e-5eb3-42fe-b466-94bcff888d92 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:52:14.520403 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f3fb26e-5eb3-42fe-b466-94bcff888d92 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:52:14.520708 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f3fb26e-5eb3-42fe-b466-94bcff888d92 admin admin] PATH_INFO: `/v3/projects/38de016089e0474d9a2b16cfecaf0a60` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:52:14.521825 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f3fb26e-5eb3-42fe-b466-94bcff888d92 admin admin] RBAC: Authorizing `identity:delete_project(project_id=38de016089e0474d9a2b16cfecaf0a60)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:52:14.523765 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f3fb26e-5eb3-42fe-b466-94bcff888d92 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:52:14.568718 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-1f3fb26e-5eb3-42fe-b466-94bcff888d92 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '38de016089e0474d9a2b16cfecaf0a60', 'request_id': 'req-1f3fb26e-5eb3-42fe-b466-94bcff888d92'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:52:14.575512 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-1f3fb26e-5eb3-42fe-b466-94bcff888d92 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '38de016089e0474d9a2b16cfecaf0a60', 'request_id': 'req-1f3fb26e-5eb3-42fe-b466-94bcff888d92'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:52:14.581081 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-1f3fb26e-5eb3-42fe-b466-94bcff888d92 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '38de016089e0474d9a2b16cfecaf0a60', 'request_id': 'req-1f3fb26e-5eb3-42fe-b466-94bcff888d92'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:52:14.591939 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-1f3fb26e-5eb3-42fe-b466-94bcff888d92 admin admin] The token cache is being invalidate because project 38de016089e0474d9a2b16cfecaf0a60 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:52:14.592229 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-1f3fb26e-5eb3-42fe-b466-94bcff888d92 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:52:14.592887 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-1f3fb26e-5eb3-42fe-b466-94bcff888d92 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:52:14.594837 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1457/2939] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:52:14 2022] DELETE /identity/v3/projects/38de016089e0474d9a2b16cfecaf0a60 => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:52:24.270006 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=35,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:52:24.588837 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=83,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:52:29.378178 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03ea5e43-372f-4dc8-9b80-971d5aaac7c2 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:52:29.379626 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-03ea5e43-372f-4dc8-9b80-971d5aaac7c2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:52:29.438896 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03ea5e43-372f-4dc8-9b80-971d5aaac7c2 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:52:29.490973 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e81ef00f-b0b4-4aa3-8d10-6d86a376f351 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:52:29.491584 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e81ef00f-b0b4-4aa3-8d10-6d86a376f351 service neutron] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:52:29.491859 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e81ef00f-b0b4-4aa3-8d10-6d86a376f351 service neutron] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:52:29.492125 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e81ef00f-b0b4-4aa3-8d10-6d86a376f351 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:52:29.493866 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-e81ef00f-b0b4-4aa3-8d10-6d86a376f351 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:52:29.505411 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e81ef00f-b0b4-4aa3-8d10-6d86a376f351 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:52:29.506836 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e81ef00f-b0b4-4aa3-8d10-6d86a376f351 service neutron] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:52:29.515471 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1483/2940] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:52:29 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 138 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:52:30.600920 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f3fb26e-5eb3-42fe-b466-94bcff888d92 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:52:30.602422 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-1f3fb26e-5eb3-42fe-b466-94bcff888d92 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:52:30.615838 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f3fb26e-5eb3-42fe-b466-94bcff888d92 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:52:30.619436 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90594b93-ef4a-4efe-a151-1135ecf174e5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:52:30.620080 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90594b93-ef4a-4efe-a151-1135ecf174e5 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:52:30.620362 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90594b93-ef4a-4efe-a151-1135ecf174e5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:52:30.620650 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90594b93-ef4a-4efe-a151-1135ecf174e5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:52:30.622436 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-90594b93-ef4a-4efe-a151-1135ecf174e5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:52:30.695805 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90594b93-ef4a-4efe-a151-1135ecf174e5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:52:30.697295 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90594b93-ef4a-4efe-a151-1135ecf174e5 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:52:30.708566 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1458/2941] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:52:30 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:52:39.511426 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=46 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:52:40.703708 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=34 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:54:01.122282 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e81ef00f-b0b4-4aa3-8d10-6d86a376f351 service neutron] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:54:01.124648 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-e81ef00f-b0b4-4aa3-8d10-6d86a376f351 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:54:01.263995 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e81ef00f-b0b4-4aa3-8d10-6d86a376f351 service neutron] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:54:01.269607 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64d0dc91-1c31-409b-a546-d0547350c0b4 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '676d04ef5a984edcaad63d9156568660', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:54:01.270215 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64d0dc91-1c31-409b-a546-d0547350c0b4 service placement] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:54:01.270526 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64d0dc91-1c31-409b-a546-d0547350c0b4 service placement] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:54:01.270798 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64d0dc91-1c31-409b-a546-d0547350c0b4 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:54:01.272543 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-64d0dc91-1c31-409b-a546-d0547350c0b4 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:54:01.284826 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64d0dc91-1c31-409b-a546-d0547350c0b4 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:54:01.286387 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64d0dc91-1c31-409b-a546-d0547350c0b4 service placement] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:54:01.297039 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1484/2942] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:54:01 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 175 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:54:11.291952 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=63 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:54:14.380100 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90594b93-ef4a-4efe-a151-1135ecf174e5 service neutron] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:54:14.381662 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-90594b93-ef4a-4efe-a151-1135ecf174e5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:54:14.422193 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90594b93-ef4a-4efe-a151-1135ecf174e5 service neutron] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:54:14.425884 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d6aacaa-3431-4465-a4e3-9d393f5f6059 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:54:14.426537 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d6aacaa-3431-4465-a4e3-9d393f5f6059 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:54:14.426818 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d6aacaa-3431-4465-a4e3-9d393f5f6059 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:54:14.427096 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d6aacaa-3431-4465-a4e3-9d393f5f6059 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:54:14.428831 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-5d6aacaa-3431-4465-a4e3-9d393f5f6059 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:54:14.489339 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d6aacaa-3431-4465-a4e3-9d393f5f6059 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:54:14.490770 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d6aacaa-3431-4465-a4e3-9d393f5f6059 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:54:14.499764 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1459/2943] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:54:14 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 120 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:54:15.565104 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64d0dc91-1c31-409b-a546-d0547350c0b4 service placement] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:54:15.569806 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-64d0dc91-1c31-409b-a546-d0547350c0b4 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:54:15.590761 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64d0dc91-1c31-409b-a546-d0547350c0b4 service placement] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:54:15.597394 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89228c60-13cd-4f21-a31d-f5340440ae0b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e703b42c53974daaa608b0ccc3f018b1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'ResellerAdmin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:54:15.598112 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89228c60-13cd-4f21-a31d-f5340440ae0b service nova] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:54:15.598453 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89228c60-13cd-4f21-a31d-f5340440ae0b service nova] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:54:15.598791 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89228c60-13cd-4f21-a31d-f5340440ae0b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:54:15.600568 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-89228c60-13cd-4f21-a31d-f5340440ae0b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:54:15.613810 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89228c60-13cd-4f21-a31d-f5340440ae0b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:54:15.615384 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89228c60-13cd-4f21-a31d-f5340440ae0b service nova] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:54:15.628140 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1485/2944] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:54:15 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 64 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:54:19.470138 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d6aacaa-3431-4465-a4e3-9d393f5f6059 service neutron] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:54:19.479624 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d6aacaa-3431-4465-a4e3-9d393f5f6059 service neutron] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:54:19.483993 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73686984-e726-45bc-99a6-912bd8e66897 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0268e88e0c5645a98bab5664be6e49bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:54:19.484541 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73686984-e726-45bc-99a6-912bd8e66897 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:54:19.484868 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73686984-e726-45bc-99a6-912bd8e66897 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:54:19.485144 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73686984-e726-45bc-99a6-912bd8e66897 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:54:19.486851 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-73686984-e726-45bc-99a6-912bd8e66897 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:54:19.571407 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73686984-e726-45bc-99a6-912bd8e66897 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:54:19.572854 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73686984-e726-45bc-99a6-912bd8e66897 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:54:19.592461 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1460/2945] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:54:19 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 123 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:54:25.621861 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=6 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:54:25.705017 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89228c60-13cd-4f21-a31d-f5340440ae0b service nova] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:54:25.720749 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89228c60-13cd-4f21-a31d-f5340440ae0b service nova] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:54:25.726191 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cc3fb4e-80db-4509-b888-3ffa24ee89cf service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '676d04ef5a984edcaad63d9156568660', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:54:25.726800 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cc3fb4e-80db-4509-b888-3ffa24ee89cf service placement] REQUEST_METHOD: `GET` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:54:25.727080 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cc3fb4e-80db-4509-b888-3ffa24ee89cf service placement] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:54:25.727550 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cc3fb4e-80db-4509-b888-3ffa24ee89cf service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:54:25.729388 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-2cc3fb4e-80db-4509-b888-3ffa24ee89cf service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:54:25.741819 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cc3fb4e-80db-4509-b888-3ffa24ee89cf service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:54:25.743403 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cc3fb4e-80db-4509-b888-3ffa24ee89cf service placement] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:54:25.755128 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1486/2946] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:54:25 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 50 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:54:25.765495 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73686984-e726-45bc-99a6-912bd8e66897 service neutron] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:54:25.767644 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-73686984-e726-45bc-99a6-912bd8e66897 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:54:25.781259 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73686984-e726-45bc-99a6-912bd8e66897 service neutron] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:54:25.786128 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4e4771d-536c-4322-8a0b-f9a31a96d818 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:54:25.786865 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e4771d-536c-4322-8a0b-f9a31a96d818 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:54:25.787218 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e4771d-536c-4322-8a0b-f9a31a96d818 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:54:25.787563 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e4771d-536c-4322-8a0b-f9a31a96d818 admin admin] PATH_INFO: `/v3/users/673b4f55a335484d927d7966d72690ad` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:54:25.789056 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4e4771d-536c-4322-8a0b-f9a31a96d818 admin admin] RBAC: Authorizing `identity:delete_user(user_id=673b4f55a335484d927d7966d72690ad)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:54:25.791058 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4e4771d-536c-4322-8a0b-f9a31a96d818 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:54:25.827053 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:54:25.841907 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-f4e4771d-536c-4322-8a0b-f9a31a96d818 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '673b4f55a335484d927d7966d72690ad'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:54:25.848858 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-f4e4771d-536c-4322-8a0b-f9a31a96d818 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '673b4f55a335484d927d7966d72690ad'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:54:25.861687 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-f4e4771d-536c-4322-8a0b-f9a31a96d818 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '673b4f55a335484d927d7966d72690ad'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:54:25.862333 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-f4e4771d-536c-4322-8a0b-f9a31a96d818 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '673b4f55a335484d927d7966d72690ad'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:54:25.869463 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-f4e4771d-536c-4322-8a0b-f9a31a96d818 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '673b4f55a335484d927d7966d72690ad'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:54:25.876067 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1461/2947] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:54:25 2022] DELETE /identity/v3/users/673b4f55a335484d927d7966d72690ad => generated 0 bytes in 111 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:54:26.049665 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cc3fb4e-80db-4509-b888-3ffa24ee89cf service placement] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:54:26.062207 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cc3fb4e-80db-4509-b888-3ffa24ee89cf service placement] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:54:26.120337 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b82c1af-52b6-47c6-ac5d-63608f94bbe3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:54:26.121055 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b82c1af-52b6-47c6-ac5d-63608f94bbe3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:54:26.121341 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b82c1af-52b6-47c6-ac5d-63608f94bbe3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:54:26.121619 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b82c1af-52b6-47c6-ac5d-63608f94bbe3 admin admin] PATH_INFO: `/v3/projects/b981336000334d7d8451feaacc75da99` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:54:26.123277 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b82c1af-52b6-47c6-ac5d-63608f94bbe3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b981336000334d7d8451feaacc75da99)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:54:26.125273 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b82c1af-52b6-47c6-ac5d-63608f94bbe3 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:54:26.173572 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-1b82c1af-52b6-47c6-ac5d-63608f94bbe3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b981336000334d7d8451feaacc75da99', 'request_id': 'req-1b82c1af-52b6-47c6-ac5d-63608f94bbe3'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:54:26.180848 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-1b82c1af-52b6-47c6-ac5d-63608f94bbe3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b981336000334d7d8451feaacc75da99', 'request_id': 'req-1b82c1af-52b6-47c6-ac5d-63608f94bbe3'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:54:26.187014 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-1b82c1af-52b6-47c6-ac5d-63608f94bbe3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b981336000334d7d8451feaacc75da99', 'request_id': 'req-1b82c1af-52b6-47c6-ac5d-63608f94bbe3'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:54:26.197749 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-1b82c1af-52b6-47c6-ac5d-63608f94bbe3 admin admin] The token cache is being invalidate because project b981336000334d7d8451feaacc75da99 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81434) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:54:26.198040 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-1b82c1af-52b6-47c6-ac5d-63608f94bbe3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:54:26.198683 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-1b82c1af-52b6-47c6-ac5d-63608f94bbe3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:54:26.200186 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1487/2948] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:54:26 2022] DELETE /identity/v3/projects/b981336000334d7d8451feaacc75da99 => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:54:26.204897 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4e4771d-536c-4322-8a0b-f9a31a96d818 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:54:26.206380 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-f4e4771d-536c-4322-8a0b-f9a31a96d818 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:54:26.219583 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4e4771d-536c-4322-8a0b-f9a31a96d818 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:54:26.274429 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d69462f5-b567-4479-b431-a80f6dc4b1b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:54:26.275050 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d69462f5-b567-4479-b431-a80f6dc4b1b1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:54:26.275330 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d69462f5-b567-4479-b431-a80f6dc4b1b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:54:26.275603 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d69462f5-b567-4479-b431-a80f6dc4b1b1 admin admin] PATH_INFO: `/v3/users/ef85cd347e6546229f92a41b74e3eb5d` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:54:26.276785 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d69462f5-b567-4479-b431-a80f6dc4b1b1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ef85cd347e6546229f92a41b74e3eb5d)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:54:26.278725 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d69462f5-b567-4479-b431-a80f6dc4b1b1 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:54:26.327241 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-d69462f5-b567-4479-b431-a80f6dc4b1b1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ef85cd347e6546229f92a41b74e3eb5d'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:54:26.334606 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-d69462f5-b567-4479-b431-a80f6dc4b1b1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ef85cd347e6546229f92a41b74e3eb5d'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:54:26.347152 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-d69462f5-b567-4479-b431-a80f6dc4b1b1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ef85cd347e6546229f92a41b74e3eb5d'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:54:26.347811 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-d69462f5-b567-4479-b431-a80f6dc4b1b1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ef85cd347e6546229f92a41b74e3eb5d'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:54:26.354863 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-d69462f5-b567-4479-b431-a80f6dc4b1b1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ef85cd347e6546229f92a41b74e3eb5d'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:54:26.360842 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1462/2949] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Sep 25 04:54:26 2022] DELETE /identity/v3/users/ef85cd347e6546229f92a41b74e3eb5d => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:54:26.530469 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b82c1af-52b6-47c6-ac5d-63608f94bbe3 admin admin] Authenticating user token {{(pid=81434) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:54:26.531994 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.fernet_utils [None req-1b82c1af-52b6-47c6-ac5d-63608f94bbe3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81434) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:54:26.546925 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b82c1af-52b6-47c6-ac5d-63608f94bbe3 admin admin] Validating token access rules against request {{(pid=81434) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:54:26.605248 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c9837d5-8561-4900-9071-6fa705b9bb97 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e61aa4e8d68f455e96668bc70e0a6679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '99bb568eb0e94976a15231a76508d22f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81434) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:54:26.605884 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c9837d5-8561-4900-9071-6fa705b9bb97 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:54:26.606167 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c9837d5-8561-4900-9071-6fa705b9bb97 admin admin] SCRIPT_NAME: `/identity` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:54:26.606443 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c9837d5-8561-4900-9071-6fa705b9bb97 admin admin] PATH_INFO: `/v3/projects/df48a331b7d444bc9fee0f035a630852` {{(pid=81434) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:54:26.608063 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c9837d5-8561-4900-9071-6fa705b9bb97 admin admin] RBAC: Authorizing `identity:delete_project(project_id=df48a331b7d444bc9fee0f035a630852)` {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:54:26.610004 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c9837d5-8561-4900-9071-6fa705b9bb97 admin admin] RBAC: Authorization granted {{(pid=81434) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:54:26.630985 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:54:26.657697 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-2c9837d5-8561-4900-9071-6fa705b9bb97 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'df48a331b7d444bc9fee0f035a630852', 'request_id': 'req-2c9837d5-8561-4900-9071-6fa705b9bb97'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:54:26.664707 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-2c9837d5-8561-4900-9071-6fa705b9bb97 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'df48a331b7d444bc9fee0f035a630852', 'request_id': 'req-2c9837d5-8561-4900-9071-6fa705b9bb97'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:54:26.670770 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-2c9837d5-8561-4900-9071-6fa705b9bb97 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'df48a331b7d444bc9fee0f035a630852', 'request_id': 'req-2c9837d5-8561-4900-9071-6fa705b9bb97'} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:54:26.681629 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-2c9837d5-8561-4900-9071-6fa705b9bb97 admin admin] The token cache is being invalidate because project df48a331b7d444bc9fee0f035a630852 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81434) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 25 04:54:26.681919 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-2c9837d5-8561-4900-9071-6fa705b9bb97 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:54:26.682558 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG keystone.notifications [None req-2c9837d5-8561-4900-9071-6fa705b9bb97 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81434) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 25 04:54:26.684053 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: [pid: 81434|app: 0|req: 1488/2950] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Sep 25 04:54:26 2022] DELETE /identity/v3/projects/df48a331b7d444bc9fee0f035a630852 => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 25 04:54:36.357783 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=71,keystone:DELETE=15,keystone:INSERT=2 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:54:36.678569 dv-cinder-855359-5-iscsi devstack@keystone.service[81434]: DEBUG dbcounter [-] [81434] Writing DB stats keystone:SELECT=16,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81434) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 25 04:59:08.131912 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d69462f5-b567-4479-b431-a80f6dc4b1b1 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 25 04:59:08.133376 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-d69462f5-b567-4479-b431-a80f6dc4b1b1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:59:08.201308 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d69462f5-b567-4479-b431-a80f6dc4b1b1 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 25 04:59:08.259107 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7b44350-9aae-4c14-8e70-4aea050961fb service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '676d04ef5a984edcaad63d9156568660', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '056ca1b1a9974a7581550de0d9217823', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 25 04:59:08.259710 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b44350-9aae-4c14-8e70-4aea050961fb service placement] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 25 04:59:08.259992 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b44350-9aae-4c14-8e70-4aea050961fb service placement] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 25 04:59:08.260265 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b44350-9aae-4c14-8e70-4aea050961fb service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 25 04:59:08.262090 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-d7b44350-9aae-4c14-8e70-4aea050961fb service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 25 04:59:08.273327 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7b44350-9aae-4c14-8e70-4aea050961fb service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 25 04:59:08.274882 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7b44350-9aae-4c14-8e70-4aea050961fb service placement] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 25 04:59:08.284649 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1463/2951] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Sep 25 04:59:08 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 153 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 25 04:59:18.280411 dv-cinder-855359-5-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=58 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}